# FSCrawler and frozen indicies

**URL:** <https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997>\
**Category:** Elasticsearch\
**Created:** [September 23, 2022, 7:19am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997 "2022-09-23T07:19:32Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [September 23, 2022, 7:19am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/1 "2022-09-23T07:19:32Z")

</div>

Hello There!  
I have downloaded ans successfully started FSCrawler for my Elasticsearch cluster.  
This is how I start an instance:

```auto
08:11:23,753 INFO [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [83.3mb/1.9gb=4.19%], RAM [130.5mb/7.7gb=1.64%], Swap [0b/0b=0.0].
08:11:23,757 INFO [f.console] No job specified. Here is the list of existing jobs:
08:11:23,764 INFO [f.console] [1] - test-job
08:11:23,764 INFO [f.console] [2] - test-job2
08:11:23,765 INFO [f.console] Choose your job [1-2]...
2
08:11:26,951 INFO [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
08:11:26,952 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
08:11:28,503 INFO [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.6
08:11:28,986 INFO [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.6
08:11:29,315 INFO [f.p.e.c.f.FsParserAbstract] FS crawler started for [test-job2] for [/mnt/fscdata/] every [1m]

```

This is my `_settings.yaml` for `test-job2`:

```auto
---
name: "test-job2"
fs:
  url: "/mnt/fscdata/"
  update_rate: "1m"
  includes:
  - "*/*.doc"
  - "*/*.pdf"
  - "*/*.txt"
  includes:
  - "*/~*"
  json_support: true
  filename_as_id: true
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: true
  index_content: true
  attributes_support: false
  raw_metadata: true
  xml_support: false
  index_folders: true
  lang_detect: true
  continue_on_error: true
  ocr:
    language: "pl"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "https://{{ip_address}}:{{port_number}}"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: false
  username: "elastic"
  password: "{{pass}}"
  ssl.certificate_authorities: "/etc/filebeat/elastic-stack-ca.pem"
  ssl.certificate: "/etc/kibana/es.crt.pem"
  ssl.key: "/etc/kibana/es.key.pem"
  index: "test-job2-new"
  index_folder: "test-job2_folder-new"

```

But i got a WARN message:

```auto
09:07:43,965 WARN [o.e.c.RestClient] request [POST https://{{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
09:07:43,974 WARN [o.e.c.RestClient] request [POST https://{{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

```

and I don't see that any file/document is ingested:

 ![obraz](https://us1.discourse-cdn.com/elastic/original/3X/c/5/c59d3805ccb18ead9638a3c0ea88234587f27cda.png)

How should I fix this? Maybe you got some ideas?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 23, 2022, 10:53am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/2 "2022-09-23T10:53:35Z")

</div>

> [@cheshirecat](#):
>
> This is how I start an instance:

Could you share the very first lines of the logs when you start (let say 30 lines)?

> [@cheshirecat](#):
>
> But i got a WARN message:

You can safely ignore this message.

> [@cheshirecat](#):
>
> and I don't see that any file/document is ingested:

Just in case, do this with:

```auto
fscrawler test-job2 --restart

```

If it still does not work, run it with:

```auto
fscrawler test-job2 --restart --debug

```

And share the full logs.

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [September 23, 2022, 11:35am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/3 "2022-09-23T11:35:12Z")

</div>

> [@dadoonet](#):
>
> Could you share the very first lines of the logs when you start (let say 30 lines)?

It looks like this:

```auto
08:12:38,182 e[32mINFO e[m [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [84.1mb/1.9gb=4.23%], RAM [231.2mb/7.7gb=2.91%], Swap [0b/0b=0.0].
08:12:39,816 e[32mINFO e[m [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
08:12:39,817 e[32mINFO e[m [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
08:12:41,358 e[32mINFO e[m [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.6
08:12:41,784 e[32mINFO e[m [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.17.6
08:12:42,077 e[32mINFO e[m [f.p.e.c.f.FsParserAbstract] FS crawler started for [test-job2] for [/mnt/fscdata/] every [1m]
08:12:42,194 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:12:42,227 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:13:42,287 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:13:42,300 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:14:42,324 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:14:42,336 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:15:42,358 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:15:42,369 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:16:42,393 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:16:42,404 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:17:42,431 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:17:42,442 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:18:42,469 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:18:42,480 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:19:42,504 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:19:42,515 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:20:42,538 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:20:42,553 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:21:42,575 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:21:42,587 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:22:42,608 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:22:42,620 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:23:42,644 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:23:42,654 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:24:42,680 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:24:42,689 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:25:42,708 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:25:42,718 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:26:42,736 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:26:42,746 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:27:42,770 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:27:42,781 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:28:42,803 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:28:42,814 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:29:42,835 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:29:42,845 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:30:42,864 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:30:42,873 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:31:42,891 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:31:42,900 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:32:42,918 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:32:42,929 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:33:42,948 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:33:42,958 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:34:42,975 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:34:42,986 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:35:43,014 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:35:43,026 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:36:43,047 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:36:43,057 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:37:43,077 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:37:43,086 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:38:43,109 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:38:43,118 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:39:43,142 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:39:43,152 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:40:43,170 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:40:43,178 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:41:43,194 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:41:43,202 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:42:43,233 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
08:42:43,246 e[33mWARN e[m [o.e.c.RestClient] request [POST {{ip_address}}:{{port_number}}/test-job2_folder-new/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-7.17.6-f65e9d338dc1d07b642e14a27f338990148ee5b6 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

```

> [@dadoonet](#):
>
> `fscrawler test-job2 --restart --debug`

I don't see any changes.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 23, 2022, 11:53am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/4 "2022-09-23T11:53:43Z")

</div>

You need to use the latest SNAPSHOT instead.

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [September 24, 2022, 5:51am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/5 "2022-09-24T05:51:44Z")

</div>

> [@dadoonet](#):
>
> You need to use the latest SNAPSHOT instead.

I think you should read your DMs 😛

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 27, 2022, 3:07am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/6 "2022-09-27T03:07:54Z")

</div>

I read it but still you need to use the latest SNAPSHOT.

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [September 27, 2022, 5:13am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/7 "2022-09-27T05:13:15Z")

</div>

Latest is? 2.10? I used it.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 27, 2022, 5:17am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/8 "2022-09-27T05:17:25Z")

</div>

In case you did not pick **the most recent** SNAPSHOT, here is the latest build as we speak: [https://s01.oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler-distribution/2.10-SNAPSHOT/fscrawler-distribution-2.10-20220927.043213-110.zip](https://s01.oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler-distribution/2.10-SNAPSHOT/fscrawler-distribution-2.10-20220927.043213-110.zip)

I highly doubt that this class still exists `f.p.e.c.f.c.v.ElasticsearchClientV7` in this build.

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [September 30, 2022, 8:48am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/9 "2022-09-30T08:48:04Z")

</div>

So I did - i took the latest SNAPSHOT version. It works pretty good. I can ingest files to elastic. One thing I can't understand: when I remove file from fs the number of file decreases but number of deleted files is still null. It looks like this:

 ![obraz](https://us1.discourse-cdn.com/elastic/original/3X/0/3/0333a937e84c679cc891e5500d86710b5e4dbadf.png)  
I created 30K files that were ingested ok. After that I deleted some files and I thought I would see a number of them here.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 30, 2022, 10:59am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/10 "2022-09-30T10:59:40Z")

</div>

Please share your FSCrawler job settings. Also are you waiting for the next run to happen?

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [September 30, 2022, 11:41am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/11 "2022-09-30T11:41:33Z")

</div>

My `_settings.yaml` is:

```auto
---
name: "test-job3"
fs:
  url: "/mnt/fscdata/"
  update_rate: "1m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "https://{{ip_address}}:{{port_number}}"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: false
  username: "elastic"
  password: "{{pass}}"
  ssl.certificate_authorities: "/etc/filebeat/elastic-stack-ca.pem"
  ssl.certificate: "/etc/kibana/es.crt.pem"
  ssl.key: "/etc/kibana/es.key.pem"

```

> [@dadoonet](#):
>
> Also are you waiting for the next run to happen?

I had FSCrawler started and running for much more than 1m (update rate). Even if I restarted it the result was same.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 30, 2022, 12:09pm UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/12 "2022-09-30T12:09:19Z")

</div>

> [@cheshirecat](#):
>
> I had FSCrawler started and running for much more than 1m (update rate). Even if I restarted it the result was same.

Ok. I hope you did not restart with `--restart` option 😉

The only way to figure this out is by running FSCrawler with the `--trace` mode.  
It's super verbose but you can see all the details, specifically date comparaisons...

May be you are hitting this here:

> <https://github.com/dadoonet/fscrawler/issues/1471>
>
> Hi David,
> 
> I just ran into the following problem using fscrawler to index file…s on a Samba fileserver for use with Spotlight and Apple clients:
> 
> when Macs copy a file to the Samba server the client will subsequently set the timestamps (atime, mtime and btime (birthtime/creation date)) of the copied file to the same value as the original file.
> 
> For older files the mtime of the files will typically be older compared to the last fscrawler invocation, as a result fscrawler will ignore this will in an index run.
> 
> Here's an example stat output from today on such a file on a Samba server:
> 
> \`\`\`
> \# stat 'some.tif'
> File: some.tif
> Size: 2416300 Blocks: 4728 IO Block: 4096 regular file
> Device: fd00h/64768d Inode: 67268640 Links: 1
> Access: (0666/-rw-rw-rw-) Uid: ( 1001/ smbtest) Gid: ( 1001/ smbtest)
> Context: system\_u:object\_r:default\_t:s0
> Access: 2022-08-03 15:36:45.779778451 +0200
> Modify: 2022-03-23 07:51:38.000000000 +0100
> Change: 2022-08-03 15:36:44.847786265 +0200
> Birth: 2022-08-03 15:36:42.485806064 +0200
> \`\`\`
> 
> As the ctime is dating back 5 months, fscrawler refused to index the file unless forced with --restart.
> 
> If fscrawler would look at the ctime value instead of mtime, this would solve this problem as the ctime can't be set by userspace and will always reflect the last date the file inode was created and possibly modified subsequently by any file content or metadata changes.
> 
> If not changing the default behaviour, would it be possible to get an fscrawler option to use ctime instead of mtime?
> 
> Thanks!
> \-slow

I'm not happy with the current implementation so I'm thinking of something else...

> <https://github.com/dadoonet/fscrawler/issues/399>
>
> According to https://docs.oracle.com/javase/tutorial/essential/io/notification.h…tml and the fact that OS supports this:
> 
> \* inotify (linux)
> \* fsevents (macos)
> \* ReadDirectoryChangesW (windows)
> 
> We can try to replace the current and super buggy implementation but there are some few things to think about.
> The first time we start, we need to index all existing data. Which means crawling manually. Then, \`loop\` option can be today any number.
> If we want to use infinite loop (-1), then it's easy. After the first crawl, just start the WatchService.
> If loop is 1, just exit after the first crawl and don't start WatchService.
> If loop is 0, don't crawl and don't start WatchService.
> 
> The problem is when loop is \> 1. What do we do?
> Also, how can we restart the WatchService from where it was stopped. May be we just need on restart to crawl again the whole directory and detect file by file, dir by dir, what has changed?
> 
> So it needs some thoughts and experiment but I believe this is the way to go.
> 
> See also #199
> 
> Some docs:
> 
> \* https://stackoverflow.com/a/33379328/1432281
> \* http://markusjais.com/file-system-events-with-java-7/

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [October 3, 2022, 9:21am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/13 "2022-10-03T09:21:06Z")

</div>

I started with:

> [@dadoonet](#):
>
> the `--trace` mode

and got a result:

```auto
11:07:25,810 ^[[32mINFO ^[[m [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [70.2mb/1.9gb=3.53%], RAM [277.6mb/7.7gb=3.49%], Swap [0b/0b=0.0].
11:07:25,813 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [6/_settings.json] already exists
11:07:25,813 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [6/_settings_folder.json] already exists
11:07:25,814 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [7/_settings.json] already exists
11:07:25,814 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [7/_settings_folder.json] already exists
11:07:25,815 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [7/_wpsearch_settings.json] already exists
11:07:25,815 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [8/_settings.json] already exists
11:07:25,816 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [8/_settings_folder.json] already exists
11:07:25,816 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] Mapping [8/_wpsearch_settings.json] already exists
11:07:25,817 ^[[36mDEBUG^[[m [f.p.e.c.f.c.FsCrawlerCli] Starting job [test-job3]...
11:07:25,818 ^[[30mTRACE^[[m [f.p.e.c.f.f.MetaFileHandler] Reading file _settings.yaml from /root/.fscrawler/test-job3
11:07:26,568 ^[[30mTRACE^[[m [f.p.e.c.f.c.FsCrawlerCli] settings used for this crawler: [---
name: "test-job3"
fs:
  url: "/mnt/fscdata/"
  update_rate: "1m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "https://{{ip_address}}:{{port_number}}"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  username: "elastic"
  ssl_verification: false
]
11:07:26,596 ^[[32mINFO ^[[m [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
11:07:26,597 ^[[32mINFO ^[[m [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
11:07:26,809 ^[[33mWARN ^[[m [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
11:07:26,872 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] get version
11:07:26,873 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] Calling GET https://{{ip_address}}:{{port_number}}/ with params []
11:07:27,856 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] GET https://{{ip_address}}:{{port_number}}/ gives {
  "name" : "node-2",
  "cluster_name" : "ks-es",
  "cluster_uuid" : "V88w5UHeSo-_KoCGSBxh3w",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

11:07:27,922 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] get version returns 7.17.6 and 7 as the major version number
11:07:27,923 ^[[32mINFO ^[[m [f.p.e.c.f.c.ElasticsearchClient] Elasticsearch Client connected to a node running version 7.17.6
11:07:27,936 ^[[36mDEBUG^[[m [f.p.e.c.f.s.FsCrawlerManagementServiceElasticsearchImpl] Elasticsearch Management Service started
11:07:27,939 ^[[33mWARN ^[[m [f.p.e.c.f.c.ElasticsearchClient] We are not doing SSL verification. It's not recommended for production.
11:07:27,946 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] get version
11:07:27,946 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] Calling GET https://{{ip_address}}:{{port_number}}/ with params []
11:07:28,092 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] GET https://{{ip_address}}:{{port_number}}/ gives {
  "name" : "node-2",
  "cluster_name" : "ks-es",
  "cluster_uuid" : "V88w5UHeSo-_KoCGSBxh3w",
  "version" : {
    "number" : "7.17.6",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "f65e9d338dc1d07b642e14a27f338990148ee5b6",
    "build_date" : "2022-08-23T11:08:48.893373482Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

11:07:28,093 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] get version returns 7.17.6 and 7 as the major version number
11:07:28,093 ^[[32mINFO ^[[m [f.p.e.c.f.c.ElasticsearchClient] Elasticsearch Client connected to a node running version 7.17.6
11:07:28,094 ^[[36mDEBUG^[[m [f.p.e.c.f.s.FsCrawlerDocumentServiceElasticsearchImpl] Elasticsearch Document Service started
11:07:28,104 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] create index [test-job3]
11:07:28,104 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] index settings: [{
  "settings": {
    "number_of_shards": 1,
    "index.mapping.total_fields.limit": 2000,
    "analysis": {
      "analyzer": {
        "fscrawler_path": {
          "tokenizer": "fscrawler_path"
        }
      },
      "tokenizer": {
        "fscrawler_path": {
          "type": "path_hierarchy"
        }
      }
    }
  },
  "mappings": {
    "dynamic_templates": [
      {
        "raw_as_text": {
          "path_match": "meta.raw.*",
          "mapping": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    ],
    "properties": {
      "attachment": {
        "type": "binary",
        "doc_values": false
      },
      "attributes": {
        "properties": {
          "group": {
            "type": "keyword"
          },
          "owner": {
            "type": "keyword"
          }
        }
      },
      "content": {
        "type": "text"
      },
      "file": {
        "properties": {
          "content_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword",
            "store": true
          },
          "extension": {
            "type": "keyword"
          },
          "filesize": {
            "type": "long"
          },
          "indexed_chars": {
            "type": "long"
          },
          "indexing_date": {
            "type": "date",
            "format": "date_optional_time"
          },
          "created": {
            "type": "date",
            "format": "date_optional_time"
          },
          "last_modified": {
            "type": "date",
            "format": "date_optional_time"
          },
          "last_accessed": {
            "type": "date",
            "format": "date_optional_time"
          },
          "checksum": {
            "type": "keyword"
          },
          "url": {
            "type": "keyword",
            "index": false
          }
        }
      },
      "meta": {
        "properties": {
          "author": {
            "type": "text"
          },
          "date": {
            "type": "date",
            "format": "date_optional_time"
          },
          "keywords": {
            "type": "text"
          },
          "title": {
            "type": "text"
          },
          "language": {
            "type": "keyword"
          },
          "format": {
            "type": "text"
          },
          "identifier": {
            "type": "text"
          },
          "contributor": {
            "type": "text"
          },
          "coverage": {
            "type": "text"
          },
          "modifier": {
            "type": "text"
          },
          "creator_tool": {
            "type": "keyword"
          },
          "publisher": {
            "type": "text"
          },
          "relation": {
            "type": "text"
          },
          "rights": {
            "type": "text"
          },
          "source": {
            "type": "text"
          },
          "type": {
            "type": "text"
          },
          "description": {
            "type": "text"
          },
          "created": {
            "type": "date",
            "format": "date_optional_time"
          },
          "print_date": {
            "type": "date",
            "format": "date_optional_time"
          },
          "metadata_date": {
            "type": "date",
            "format": "date_optional_time"
          },
          "latitude": {
            "type": "text"
          },
          "longitude": {
            "type": "text"
          },
          "altitude": {
            "type": "text"
          },
          "rating": {
            "type": "byte"
          },
          "comments": {
            "type": "text"
          }
        }
      },
      "path": {
        "properties": {
          "real": {
            "type": "keyword",
            "fields": {
              "tree": {
                "type": "text",
                "analyzer": "fscrawler_path",
                "fielddata": true
              },
              "fulltext": {
                "type": "text"
              }
            }
          },
          "root": {
            "type": "keyword"
          },
          "virtual": {
            "type": "keyword",
            "fields": {
              "tree": {
                "type": "text",
                "analyzer": "fscrawler_path",
                "fielddata": true
              },
              "fulltext": {
                "type": "text"
              }
            }
          }
        }
      }
    }
  }
}
]
11:07:28,105 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] Calling PUT https://{{ip_address}}:{{port_number}}/test-job3 with params []
11:07:28,136 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] Error while running PUT https://{{ip_address}}:{{port_number}}/test-job3: {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [test-job3/NLNLo2o9SUCbLkkzmmHBJg] already exists","index_uuid":"NLNLo2o9SUCbLkkzmmHBJg","index":"test-job3"}],"type":"resource_already_exists_exception","reason":"index [test-job3/NLNLo2o9SUCbLkkzmmHBJg] already exists","index_uuid":"NLNLo2o9SUCbLkkzmmHBJg","index":"test-job3"},"status":400}
11:07:28,136 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] Response for create index [test-job3]: HTTP 400 Bad Request
11:07:28,138 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] create index [test-job3_folder]
11:07:28,138 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] index settings: [{
  "settings": {
    "analysis": {
      "analyzer": {
        "fscrawler_path": {
          "tokenizer": "fscrawler_path"
        }
      },
      "tokenizer": {
        "fscrawler_path": {
          "type": "path_hierarchy"
        }
      }
    }
  },
  "mappings": {
    "properties" : {
      "file": {
        "properties": {
          "content_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword",
            "store": true
          }
        }
      },
      "path": {
        "properties": {
          "real": {
            "type": "keyword",
            "fields": {
              "tree": {
                "type": "text",
                "analyzer": "fscrawler_path",
                "fielddata": true
              },
              "fulltext": {
                "type": "text"
              }
            }
          },
          "root": {
            "type": "keyword"
          },
          "virtual": {
            "type": "keyword",
            "fields": {
              "tree": {
                "type": "text",
                "analyzer": "fscrawler_path",
                "fielddata": true
              },
              "fulltext": {
                "type": "text"
              }
            }
          }
        }
      }
    }
  }
}
]
11:07:28,138 ^[[30mTRACE^[[m [f.p.e.c.f.c.ElasticsearchClient] Calling PUT https://{{ip_address}}:{{port_number}}/test-job3_folder with params []
11:07:28,149 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] Error while running PUT https://{{ip_address}}:{{port_number}}/test-job3_folder: {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [test-job3_folder/190Zc46hS1KXVbw4-kwfTw] already exists","index_uuid":"190Zc46hS1KXVbw4-kwfTw","index":"test-job3_folder"}],"type":"resource_already_exists_exception","reason":"index [test-job3_folder/190Zc46hS1KXVbw4-kwfTw] already exists","index_uuid":"190Zc46hS1KXVbw4-kwfTw","index":"test-job3_folder"},"status":400}
11:07:28,149 ^[[36mDEBUG^[[m [f.p.e.c.f.c.ElasticsearchClient] Response for create index [test-job3_folder]: HTTP 400 Bad Request
11:07:28,155 ^[[36mDEBUG^[[m [f.p.e.c.f.FsParserAbstract] creating fs crawler thread [test-job3] for [/mnt/fscdata/] every [1m]
11:07:28,156 ^[[32mINFO ^[[m [f.p.e.c.f.FsParserAbstract] FS crawler started for [test-job3] for [/mnt/fscdata/] every [1m]
11:07:28,162 ^[[36mDEBUG^[[m [f.p.e.c.f.FsParserAbstract] Fs crawler thread [test-job3] is now running. Run #1...
11:07:28,199 ^[[30mTRACE^[[m [f.p.e.c.f.f.MetaFileHandler] Reading file _status.json from /root/.fscrawler/test-job3
11:07:28,217 ^[[36mDEBUG^[[m [f.p.e.c.f.FsParserAbstract] indexing [/mnt/fscdata/] content
11:07:28,218 ^[[36mDEBUG^[[m [f.p.e.c.f.c.f.FileAbstractorFile] Listing local files from /mnt/fscdata/
11:07:32,162 ^[[36mDEBUG^[[m [f.p.e.c.f.c.f.FileAbstractorFile] 28900 local files found
11:07:32,207 ^[[30mTRACE^[[m [f.p.e.c.f.FsParserAbstract] FileAbstractModel = FileAbstractModel{name='dokument25536.txt', file=true, directory=false, lastModifiedDate=2022-09-28T14:56:57.128069, creationDate=2022-09-28T14:56:57.128069, accessDate=2022-09-28T14:57:56.410777, path='/mnt/fscdata/', owner='root', group='root', permissions=644, extension='txt', fullpath='/mnt/fscdata/dokument25536.txt', size=1024}
11:07:32,208 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] computeVirtualPathName(/mnt/fscdata/, /mnt/fscdata/dokument25536.txt) = dokument25536.txt
11:07:32,208 ^[[36mDEBUG^[[m [f.p.e.c.f.f.FsCrawlerUtil] directory = [false], filename = [dokument25536.txt], includes = [null], excludes = [[*/~*]]

```

---

<div class="post-metadata">

**Author:** ![cheshirecat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheshirecat/32/109532_2.png) [@cheshirecat](https://discuss.elastic.co/u/cheshirecat)\
**Post date:** [October 5, 2022, 8:27am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/14 "2022-10-05T08:27:21Z")

</div>

After a try with `--restart` I got an error with SANs that we managed to resolve [here](https://discuss.elastic.co/t/fscrawler-ssl-sans/314263/27)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 2, 2022, 8:28am UTC](https://discuss.elastic.co/t/fscrawler-and-frozen-indicies/314997/15 "2022-11-02T08:28:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.