Looking to migrate away from docker and to the installed application

[ScottCov]

I have been using docker which is I think fscrawler 2.10 which has yielded very good results and I am looking to migrate away from the container to installed application (also 2.10). I want to be absolutely sure that I get the same results as the docker version. I installed tesseract. Are there any other libraries I need to install like PDFBox? If I am not expecting JPX/JP2 images is there any reason to install JPEG 2000? I am pretty sure I am missing something because the installed version is indexing at 3-4X the rate the docker version did...I am getting these warnings:

10:43:25,629 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif-Italic for TimesNewRomanPS-ItalicMT
10:43:25,630 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for TimesNewRomanPSMT
10:43:25,630 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for CourierNewPSMT
10:43:25,631 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif-Bold for TimesNewRomanPS-BoldMT
10:43:25,634 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif-Italic for TimesNewRomanPS-ItalicMT
10:43:25,723 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for TimesNewRomanPSMT
10:43:25,724 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif-Bold for TimesNewRomanPS-BoldMT
10:43:25,726 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for CourierNewPSMT
10:43:25,728 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif-Italic for TimesNewRomanPS-ItalicMT
10:43:25,728 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for TimesNewRomanPSMT
10:43:25,729 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for CourierNewPSMT
10:43:25,729 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif-Bold for TimesNewRomanPS-BoldMT
10:43:25,731 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif-Italic for TimesNewRomanPS-ItalicMT
10:43:25,731 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for TimesNewRomanPSMT
10:43:25,732 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for CourierNewPSMT
10:43:25,732 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif-Bold for TimesNewRomanPS-BoldMT
10:43:25,737 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif-Italic for TimesNewRomanPS-ItalicMT
10:43:25,818 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSans for Verdana
10:43:25,818 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for TimesNewRomanPSMT
10:43:25,818 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSans-Bold for Verdana-Bold
10:43:25,822 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for CourierNewPSMT
10:43:25,822 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSans-Italic for Verdana-Italic
10:43:25,822 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSans for Verdana
10:43:25,823 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for TimesNewRomanPSMT
10:43:25,823 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSans-Bold for Verdana-Bold
10:43:25,824 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for CourierNewPSMT
10:43:25,825 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSans-Italic for Verdana-Italic
10:43:25,825 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSans for Verdana
10:43:25,826 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for TimesNewRomanPSMT
10:43:25,826 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSerif for CourierNewPSMT
10:43:25,826 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSans-Bold for Verdana-Bold
10:43:25,829 WARN [o.a.p.p.f.PDTrueTypeFont] Using fallback font LiberationSans-Italic for Verdana-Italic

[dadoonet]

Hey!

I installed tesseract.

Great. It's needed for OCR. Make sure it's part of your PATH otherwise it will be ignored and FSCrawler won't do OCR.

Are there any other libraries I need to install like PDFBox?

Nope. All the rest should be built-in.

If I am not expecting JPX/JP2 images is there any reason to install JPEG 2000?

No. I don't think it's needed. But I'm unsure though :)

I am pretty sure I am missing something because the installed version is indexing at 3-4X the rate the docker version did...

That's a great improvement! The only reason I could see would be that OCR is not called? Or may be OCR within the Docker image is too slow?
Do you have the same FSCrawler job settings with and without the docker image?

[ScottCov]

The only setting I changed were the fs:url and ocr path/data path. I also had to get rid of node under elasticsearch and just specify the url

name: "job_name_new"
fs:
  url: "/mnt/cloud/cases"
  #url: "/tmp/es"
  update_rate: "15m"
  excludes:
  - "*/~*"
  json_support: false
  #indexed_chars: "-1"
  filename_as_id: true
  add_filesize: true
  remove_deleted: false
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: true
  ocr:
    path: "/usr/bin/tesseract"
    data_path: "/usr/share/tesseract-ocr/5/tessdata/"
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  pipeline: "fscrawler-copy"
  urls:
  - "https://192.168.1.201:9200"
  username: "elastic"
  password: "xxxxxxxxxxxxxxx"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: false

But it looks like OCR isn't running???

11:03:51,315 INFO [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [357.4mb/5.8gb=5.95%], RAM [8.7gb/23.4gb=37.17%], Swap [3.7gb/3.7gb=100.0%].
11:03:51,767 WARN [f.p.e.c.f.c.FsCrawlerCli] username/password is deprecated. Use apiKey instead.
11:03:51,827 WARN [f.p.e.c.f.FsCrawlerImpl] username/password is deprecated. Use apiKey instead.
11:03:52,635 WARN [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
11:03:52,646 WARN [f.p.e.c.f.c.ElasticsearchClient] Semantic search is enabled but we are running Elasticsearch with a basic license although we need either an enterprise or trial license.We will not be able to use the semantic search features ATM. We might switch later to a vector embeddings generation.
11:03:52,771 WARN [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
11:03:52,781 WARN [f.p.e.c.f.c.ElasticsearchClient] Semantic search is enabled but we are running Elasticsearch with a basic license although we need either an enterprise or trial license.We will not be able to use the semantic search features ATM. We might switch later to a vector embeddings generation.
11:03:52,879 INFO [f.p.e.c.f.FsCrawlerImpl] FSCrawler is now connected to Elasticsearch version [8.17.4]
11:03:52,880 INFO [f.p.e.c.f.FsCrawlerImpl] FSCrawler started in watch mode. It will run unless you stop it with CTRL+C.
11:03:52,881 INFO [f.p.e.c.f.FsParserAbstract] FS crawler started for [job_name_new] for [/mnt/cloud/cases] every [15m]
11:03:52,881 INFO [f.p.e.c.f.FsParserAbstract] Run #1: job [job_name_new]: starting...
11:03:53,423 INFO [f.p.e.c.f.t.TikaInstance] OCR is disabled.

[dadoonet]

11:03:53,423 INFO [f.p.e.c.f.t.TikaInstance] OCR is disabled.

I think that's the reason it's so fast. No OCR apparently.

Could run in debug mode to check what is happening when it starts?

[ScottCov]

serveracct@pauli:~$ which tesseract
/usr/bin/tesseract

Yet getting message directory does not exist?

11:38:42,365 DEBUG [f.p.e.c.f.t.TikaInstance] Tesseract Path set to [/usr/bin/tesseract].
11:38:42,365 DEBUG [f.p.e.c.f.t.TikaInstance] Tesseract Data Path set to [/usr/share/tesseract-ocr/5/tessdata/].
11:38:42,366 DEBUG [f.p.e.c.f.t.TikaInstance] Tesseract is not correctly set up so we won't run OCR. Error is: tesseractPath (/usr/bin/tesseract/) doesn't point to an existing directory
11:38:42,366 DEBUG [f.p.e.c.f.t.TikaInstance] Fullstack trace error for Tesseract
org.apache.tika.exception.TikaConfigException: tesseractPath (/usr/bin/tesseract/) doesn't point to an existing directory

[dadoonet]

Could you try with /usr/bin instead?

[ScottCov]

What about data path? Change that to "/usr/share/tesseract-ocr/5?
ocr:
path: "/usr/bin/tesseract"
data_path: "/usr/share/tesseract-ocr/5/tessdata/"

[dadoonet]

No I think that's ok. See https://fscrawler.readthedocs.io/en/latest/user/ocr.html#ocr-data-path.

What the logs are telling now?