FSCrawler and frozen indicies

cheshirecat · September 23, 2022, 7:19am

Hello There!
I have downloaded ans successfully started FSCrawler for my Elasticsearch cluster.
This is how I start an instance:

08:11:23,753 INFO  [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [83.3mb/1.9gb=4.19%], RAM [130.5mb/7.7gb=1.64%], Swap [0b/0b=0.0].
08:11:23,757 INFO  [f.console] No job specified. Here is the list of existing jobs:
08:11:23,764 INFO  [f.console] [1] - test-job
08:11:23,764 INFO  [f.console] [2] - test-job2
08:11:23,765 INFO  [f.console] Choose your job [1-2]...
2
08:11:26,951 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
08:11:26,952 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
08:11:28,503 INFO  [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.6
08:11:28,986 INFO  [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.6
08:11:29,315 INFO  [f.p.e.c.f.FsParserAbstract] FS crawler started for [test-job2] for [/mnt/fscdata/] every [1m]

This is my _settings.yaml for test-job2:

---
name: "test-job2"
fs:
  url: "/mnt/fscdata/"
  update_rate: "1m"
  includes:
  - "*/*.doc"
  - "*/*.pdf"
  - "*/*.txt"
  includes:
  - "*/~*"
  json_support: true
  filename_as_id: true
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: true
  index_content: true
  attributes_support: false
  raw_metadata: true
  xml_support: false
  index_folders: true
  lang_detect: true
  continue_on_error: true
  ocr:
    language: "pl"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "https://{{ip_address}}:{{port_number}}"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: false
  username: "elastic"
  password: "{{pass}}"
  ssl.certificate_authorities: "/etc/filebeat/elastic-stack-ca.pem"
  ssl.certificate: "/etc/kibana/es.crt.pem"
  ssl.key: "/etc/kibana/es.key.pem"
  index: "test-job2-new"
  index_folder: "test-job2_folder-new"

But i got a WARN message:

09:07:43,965 WARN  [o.e.c.RestClient] request [POST https://{{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
09:07:43,974 WARN  [o.e.c.RestClient] request [POST https://{{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

and I don't see that any file/document is ingested:

How should I fix this? Maybe you got some ideas?

dadoonet · September 23, 2022, 10:53am

Could you share the very first lines of the logs when you start (let say 30 lines)?

You can safely ignore this message.

Just in case, do this with:

fscrawler test-job2 --restart

If it still does not work, run it with:

fscrawler test-job2 --restart --debug

And share the full logs.

cheshirecat · September 23, 2022, 11:35am

It looks like this:

08:12:38,182 e[32mINFO e[m [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [84.1mb/1.9gb=4.23%], RAM [231.2mb/7.7gb=2.91%], Swap [0b/0b=0.0].
08:12:39,816 e[32mINFO e[m [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
08:12:39,817 e[32mINFO e[m [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
08:12:41,358 e[32mINFO e[m [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.6
08:12:41,784 e[32mINFO e[m [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.6
08:12:42,077 e[32mINFO e[m [f.p.e.c.f.FsParserAbstract] FS crawler started for [test-job2] for [/mnt/fscdata/] every [1m]
08:12:42,194 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:12:42,227 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:13:42,287 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:13:42,300 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:14:42,324 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:14:42,336 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:15:42,358 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:15:42,369 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:16:42,393 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:16:42,404 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:17:42,431 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:17:42,442 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:18:42,469 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:18:42,480 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:19:42,504 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:19:42,515 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:20:42,538 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:20:42,553 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:21:42,575 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:21:42,587 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:22:42,608 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:22:42,620 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:23:42,644 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:23:42,654 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:24:42,680 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:24:42,689 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:25:42,708 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:25:42,718 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:26:42,736 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:26:42,746 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:27:42,770 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:27:42,781 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:28:42,803 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:28:42,814 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:29:42,835 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:29:42,845 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:30:42,864 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:30:42,873 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:31:42,891 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:31:42,900 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:32:42,918 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:32:42,929 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:33:42,948 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:33:42,958 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:34:42,975 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:34:42,986 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:35:43,014 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:35:43,026 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:36:43,047 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:36:43,057 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:37:43,077 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:37:43,086 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:38:43,109 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:38:43,118 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:39:43,142 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:39:43,152 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:40:43,170 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:40:43,178 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:41:43,194 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:41:43,202 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:42:43,233 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:42:43,246 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

I don't see any changes.

dadoonet · September 23, 2022, 11:53am

You need to use the latest SNAPSHOT instead.

cheshirecat · September 24, 2022, 5:51am

I think you should read your DMs

dadoonet · September 27, 2022, 3:07am

I read it but still you need to use the latest SNAPSHOT.

cheshirecat · September 27, 2022, 5:13am

Latest is? 2.10? I used it.

dadoonet · September 27, 2022, 5:17am

In case you did not pick the most recent SNAPSHOT, here is the latest build as we speak: https://s01.oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler-distribution/2.10-SNAPSHOT/fscrawler-distribution-2.10-20220927.043213-110.zip

I highly doubt that this class still exists f.p.e.c.f.c.v.ElasticsearchClientV7 in this build.

cheshirecat · September 30, 2022, 8:48am

So I did - i took the latest SNAPSHOT version. It works pretty good. I can ingest files to elastic. One thing I can't understand: when I remove file from fs the number of file decreases but number of deleted files is still null. It looks like this:

I created 30K files that were ingested ok. After that I deleted some files and I thought I would see a number of them here.

dadoonet · September 30, 2022, 10:59am

Please share your FSCrawler job settings. Also are you waiting for the next run to happen?

cheshirecat · September 30, 2022, 11:41am

My _settings.yaml is:

---
name: "test-job3"
fs:
  url: "/mnt/fscdata/"
  update_rate: "1m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "https://{{ip_address}}:{{port_number}}"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: false
  username: "elastic"
  password: "{{pass}}"
  ssl.certificate_authorities: "/etc/filebeat/elastic-stack-ca.pem"
  ssl.certificate: "/etc/kibana/es.crt.pem"
  ssl.key: "/etc/kibana/es.key.pem"

I had FSCrawler started and running for much more than 1m (update rate). Even if I restarted it the result was same.

dadoonet · September 30, 2022, 12:09pm

Ok. I hope you did not restart with --restart option

The only way to figure this out is by running FSCrawler with the --trace mode.
It's super verbose but you can see all the details, specifically date comparaisons...

May be you are hitting this here:

github.com/dadoonet/fscrawler

Use file ctime, not mtime for detecting modified or new files

opened 02:56PM - 03 Aug 22 UTC

slowfranklin

feature_request component:core

Hi David, I just ran into the following problem using fscrawler to index file…s on a Samba fileserver for use with Spotlight and Apple clients: when Macs copy a file to the Samba server the client will subsequently set the timestamps (atime, mtime and btime (birthtime/creation date)) of the copied file to the same value as the original file. For older files the mtime of the files will typically be older compared to the last fscrawler invocation, as a result fscrawler will ignore this will in an index run. Here's an example stat output from today on such a file on a Samba server: ``` # stat 'some.tif' File: some.tif Size: 2416300 Blocks: 4728 IO Block: 4096 regular file Device: fd00h/64768d Inode: 67268640 Links: 1 Access: (0666/-rw-rw-rw-) Uid: ( 1001/ smbtest) Gid: ( 1001/ smbtest) Context: system_u:object_r:default_t:s0 Access: 2022-08-03 15:36:45.779778451 +0200 Modify: 2022-03-23 07:51:38.000000000 +0100 Change: 2022-08-03 15:36:44.847786265 +0200 Birth: 2022-08-03 15:36:42.485806064 +0200 ``` As the ctime is dating back 5 months, fscrawler refused to index the file unless forced with --restart. If fscrawler would look at the ctime value instead of mtime, this would solve this problem as the ctime can't be set by userspace and will always reflect the last date the file inode was created and possibly modified subsequently by any file content or metadata changes. If not changing the default behaviour, would it be possible to get an fscrawler option to use ctime instead of mtime? Thanks! -slow

I'm not happy with the current implementation so I'm thinking of something else...

github.com/dadoonet/fscrawler

Use a WatchService implementation

opened 09:24AM - 05 Jul 17 UTC

dadoonet

update

According to https://docs.oracle.com/javase/tutorial/essential/io/notification.h…tml and the fact that OS supports this: * inotify (linux) * fsevents (macos) * ReadDirectoryChangesW (windows) We can try to replace the current and super buggy implementation but there are some few things to think about. The first time we start, we need to index all existing data. Which means crawling manually. Then, `loop` option can be today any number. If we want to use infinite loop (-1), then it's easy. After the first crawl, just start the WatchService. If loop is 1, just exit after the first crawl and don't start WatchService. If loop is 0, don't crawl and don't start WatchService. The problem is when loop is > 1. What do we do? Also, how can we restart the WatchService from where it was stopped. May be we just need on restart to crawl again the whole directory and detect file by file, dir by dir, what has changed? So it needs some thoughts and experiment but I believe this is the way to go. See also #199 Some docs: * https://stackoverflow.com/a/33379328/1432281 * http://markusjais.com/file-system-events-with-java-7/

cheshirecat · October 3, 2022, 9:21am

I started with:

and got a result:

11:07:25,810 ^[[32mINFO ^[[m [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [70.2mb/1.9gb=3.53%], RAM [277.6mb/7.7gb=3.49%], Swap [0b/0b=0.0].
11:07:25,813 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [6/_settings.json] already exists
11:07:25,813 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [6/_settings_folder.json] already exists
11:07:25,814 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [7/_settings.json] already exists
11:07:25,814 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [7/_settings_folder.json] already exists
11:07:25,815 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [7/_wpsearch_settings.json] already exists
11:07:25,815 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [8/_settings.json] already exists
11:07:25,816 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [8/_settings_folder.json] already exists
11:07:25,816 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [8/_wpsearch_settings.json] already exists
11:07:25,817 ^[[36mDEBUG^[[m [f.p.e.c.f.c.FsCrawlerCli] Starting job [test-job3]...
11:07:25,818 ^[[30mTRACE^[[m [f.p.e.c.f.f.MetaFileHandler] Reading file _settings.yaml from /root/.fscrawler/test-job3
11:07:26,568 ^[[30mTRACE^[[m [f.p.e.c.f.c.FsCrawlerCli] settings used for this crawler: [---
name: "test-job3"
fs:
  url: "/mnt/fscdata/"
  update_rate: "1m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "https://{{ip_address}}:{{port_number}}"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  username: "elastic"
  ssl_verification: false
]
11:07:26,596 ^[[32mINFO ^[[m [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
11:07:26,597 ^[[32mINFO ^[[m [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
11:07:26,809 ^[[33mWARN ^[[m [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
11:07:26,872 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] get version
11:07:26,873 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] Calling GET https://{{ip_address}}:{{port_number}}/ with params []
11:07:27,856 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] GET https://{{ip_address}}:{{port_number}}/ gives {
  "name" : "node-2",
  "cluster_name" : "ks-es",
  "cluster_uuid" : "V88w5UHeSo-_KoCGSBxh3w",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

11:07:27,922 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] get version returns 7.17.6 and 7 as the major version number
11:07:27,923 ^[[32mINFO ^[[m [f.p.e.c.f.c.ElasticsearchClient] Elasticsearch Client connected to a node running version 7.17.6
11:07:27,936 ^[[36mDEBUG^[[m [f.p.e.c.f.s.FsCrawlerManagementServiceElasticsearchImpl] Elasticsearch Management Service started
11:07:27,939 ^[[33mWARN ^[[m [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
11:07:27,946 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] get version
11:07:27,946 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] Calling GET https://{{ip_address}}:{{port_number}}/ with params []
11:07:28,092 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] GET https://{{ip_address}}:{{port_number}}/ gives {
  "name" : "node-2",
  "cluster_name" : "ks-es",
  "cluster_uuid" : "V88w5UHeSo-_KoCGSBxh3w",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

11:07:28,093 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] get version returns 7.17.6 and 7 as the major version number
11:07:28,093 ^[[32mINFO ^[[m [f.p.e.c.f.c.ElasticsearchClient] Elasticsearch Client connected to a node running version 7.17.6
11:07:28,094 ^[[36mDEBUG^[[m [f.p.e.c.f.s.FsCrawlerDocumentServiceElasticsearchImpl] Elasticsearch Document Service started
11:07:28,104 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] create index [test-job3]
11:07:28,104 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] index settings: [{
  "settings": {
    "number_of_shards": 1,
    "index.mapping.total_fields.limit": 2000,
    "analysis": {
      "analyzer": {
        "fscrawler_path": {
          "tokenizer": "fscrawler_path"
        }
      },
      "tokenizer": {
        "fscrawler_path": {
          "type": "path_hierarchy"
        }
      }
    }
  },
  "mappings": {
    "dynamic_templates": [
      {
        "raw_as_text": {
          "path_match": "meta.raw.*",
          "mapping": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    ],
    "properties": {
      "attachment": {
        "type": "binary",
        "doc_values": false
      },
      "attributes": {
        "properties": {
          "group": {
            "type": "keyword"
          },
          "owner": {
            "type": "keyword"
          }
        }
      },
      "content": {
        "type": "text"
      },
      "file": {
        "properties": {
          "content_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword",
            "store": true
          },
          "extension": {
            "type": "keyword"
          },
          "filesize": {
            "type": "long"
          },
          "indexed_chars": {
            "type": "long"
          },
          "indexing_date": {
            "type": "date",
            "format": "date_optional_time"
          },
          "created": {
            "type": "date",
            "format": "date_optional_time"
          },
          "last_modified": {
            "type": "date",
            "format": "date_optional_time"
          },
          "last_accessed": {
            "type": "date",
            "format": "date_optional_time"
          },
          "checksum": {
            "type": "keyword"
          },
          "url": {
            "type": "keyword",
            "index": false
          }
        }
      },
      "meta": {
        "properties": {
          "author": {
            "type": "text"
          },
          "date": {
            "type": "date",
            "format": "date_optional_time"
          },
          "keywords": {
            "type": "text"
          },
          "title": {
            "type": "text"
          },
          "language": {
            "type": "keyword"
          },
          "format": {
            "type": "text"
          },
          "identifier": {
            "type": "text"
          },
          "contributor": {
            "type": "text"
          },
          "coverage": {
            "type": "text"
          },
          "modifier": {
            "type": "text"
          },
          "creator_tool": {
            "type": "keyword"
          },
          "publisher": {
            "type": "text"
          },
          "relation": {
            "type": "text"
          },
          "rights": {
            "type": "text"
          },
          "source": {
            "type": "text"
          },
          "type": {
            "type": "text"
          },
          "description": {
            "type": "text"
          },
          "created": {
            "type": "date",
            "format": "date_optional_time"
          },
          "print_date": {
            "type": "date",
            "format": "date_optional_time"
          },
          "metadata_date": {
            "type": "date",
            "format": "date_optional_time"
          },
          "latitude": {
            "type": "text"
          },
          "longitude": {
            "type": "text"
          },
          "altitude": {
            "type": "text"
          },
          "rating": {
            "type": "byte"
          },
          "comments": {
            "type": "text"
          }
        }
      },
      "path": {
        "properties": {
          "real": {
            "type": "keyword",
            "fields": {
              "tree": {
                "type": "text",
                "analyzer": "fscrawler_path",
                "fielddata": true
              },
              "fulltext": {
                "type": "text"
              }
            }
          },
          "root": {
            "type": "keyword"
          },
          "virtual": {
            "type": "keyword",
            "fields": {
              "tree": {
                "type": "text",
                "analyzer": "fscrawler_path",
                "fielddata": true
              },
              "fulltext": {
                "type": "text"
              }
            }
          }
        }
      }
    }
  }
}
]
11:07:28,105 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] Calling PUT https://{{ip_address}}:{{port_number}}/test-job3 with params []
11:07:28,136 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] Error while running PUT https://{{ip_address}}:{{port_number}}/test-job3: {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [test-job3/NLNLo2o9SUCbLkkzmmHBJg] already exists","index_uuid":"NLNLo2o9SUCbLkkzmmHBJg","index":"test-job3"}],"type":"resource_already_exists_exception","reason":"index [test-job3/NLNLo2o9SUCbLkkzmmHBJg] already exists","index_uuid":"NLNLo2o9SUCbLkkzmmHBJg","index":"test-job3"},"status":400}
11:07:28,136 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] Response for create index [test-job3]: HTTP 400 Bad Request
11:07:28,138 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] create index [test-job3_folder]
11:07:28,138 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] index settings: [{
  "settings": {
    "analysis": {
      "analyzer": {
        "fscrawler_path": {
          "tokenizer": "fscrawler_path"
        }
      },
      "tokenizer": {
        "fscrawler_path": {
          "type": "path_hierarchy"
        }
      }
    }
  },
  "mappings": {
    "properties" : {
      "file": {
        "properties": {
          "content_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword",
            "store": true
          }
        }
      },
      "path": {
        "properties": {
          "real": {
            "type": "keyword",
            "fields": {
              "tree": {
                "type": "text",
                "analyzer": "fscrawler_path",
                "fielddata": true
              },
              "fulltext": {
                "type": "text"
              }
            }
          },
          "root": {
            "type": "keyword"
          },
          "virtual": {
            "type": "keyword",
            "fields": {
              "tree": {
                "type": "text",
                "analyzer": "fscrawler_path",
                "fielddata": true
              },
              "fulltext": {
                "type": "text"
              }
            }
          }
        }
      }
    }
  }
}
]
11:07:28,138 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] Calling PUT https://{{ip_address}}:{{port_number}}/test-job3_folder with params []
11:07:28,149 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] Error while running PUT https://{{ip_address}}:{{port_number}}/test-job3_folder: {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [test-job3_folder/190Zc46hS1KXVbw4-kwfTw] already exists","index_uuid":"190Zc46hS1KXVbw4-kwfTw","index":"test-job3_folder"}],"type":"resource_already_exists_exception","reason":"index [test-job3_folder/190Zc46hS1KXVbw4-kwfTw] already exists","index_uuid":"190Zc46hS1KXVbw4-kwfTw","index":"test-job3_folder"},"status":400}
11:07:28,149 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] Response for create index [test-job3_folder]: HTTP 400 Bad Request
11:07:28,155 ^[[36mDEBUG^[[m [f.p.e.c.f.FsParserAbstract] creating fs crawler thread [test-job3] for [/mnt/fscdata/] every [1m]
11:07:28,156 ^[[32mINFO ^[[m [f.p.e.c.f.FsParserAbstract] FS crawler started for [test-job3] for [/mnt/fscdata/] every [1m]
11:07:28,162 ^[[36mDEBUG^[[m [f.p.e.c.f.FsParserAbstract] Fs crawler thread [test-job3] is now running. Run #1...
11:07:28,199 ^[[30mTRACE^[[m [f.p.e.c.f.f.MetaFileHandler] Reading file _status.json from /root/.fscrawler/test-job3
11:07:28,217 ^[[36mDEBUG^[[m [f.p.e.c.f.FsParserAbstract] indexing [/mnt/fscdata/] content
11:07:28,218 ^[[36mDEBUG^[[m [f.p.e.c.f.c.f.FileAbstractorFile] Listing local files from /mnt/fscdata/
11:07:32,162 ^[[36mDEBUG^[[m [f.p.e.c.f.c.f.FileAbstractorFile] 28900 local files found
11:07:32,207 ^[[30mTRACE^[[m [f.p.e.c.f.FsParserAbstract] FileAbstractModel = FileAbstractModel{name='dokument25536.txt', file=true, directory=false, lastModifiedDate=2022-09-28T14:56:57.128069, creationDate=2022-09-28T14:56:57.128069, accessDate=2022-09-28T14:57:56.410777, path='/mnt/fscdata/', owner='root', group='root', permissions=644, extension='txt', fullpath='/mnt/fscdata/dokument25536.txt', size=1024}
11:07:32,208 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/mnt/fscdata/, /mnt/fscdata/dokument25536.txt) = dokument25536.txt
11:07:32,208 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] directory = [false], filename = [dokument25536.txt], includes = [null], excludes = [[*/~*]]

cheshirecat · October 5, 2022, 8:27am

After a try with --restart I got an error with SANs that we managed to resolve here

system · November 2, 2022, 8:28am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.