Hello
I have created three document collections with which to experiment with Elastic (7.15)
Each collection is stored in a separate folder.
The folders are named: papers, numb3rs & novels.
I have used FS Crawler 2.7 to create an index for each collection.
The _settings.yaml file was essentially the same for each collection, see below.
Note - I am interested in searching text wherever it may appear in a document and therefore requires a minimal schema.
After running FS Crawler, the Kibana Index Manager displayed the 'papers' & 'numb3rs' results as expected, but zero for the 'novels'
FS Crawler appeared to work the same for all three collections.
How can I resolve this?
Regards
David
full report included below, but the 'FATAL' line is here:
16:59:39,745 FATAL [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=cluster_block_exception, reason=index [novels] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]
But I don't know what it means or what I should do to resolve it.
In full:
root@volume1:/media/fscrawler# bin/fscrawler /media/fscrawler/novels --restart -- --debug
16:48:34,317 INFO [f.console] ,----------------------------------------------------------------------------------------------------.
| ,---,. .--.--. ,----.. ,--, 2.7 |
| ,' .' | / / '. / / \ ,--.'| |
| ,---.' || : /`. / | : : __ ,-. .---.| | : __ ,-. |
| | | .'; | |--` . | ;. /,' ,'/ /| /. ./|: : ' ,' ,'/ /| |
| : : : | : ;_ . ; /--` ' | |' | ,--.--. .-'-. ' || ' | ,---. ' | |' | |
| : | |-, \ \ `. ; | ; | | ,'/ \ /___/ \: |' | | / \ | | ,' |
| | : ;/| `----. \| : | ' : / .--. .-. | .-'.. ' ' .| | : / / |' : / |
| | | .' __ \ \ |. | '___ | | ' \__\/: . ./___/ \: '' : |__ . ' / || | ' |
| ' : ' / /`--' /' ; : .'|; : | ," .--.; |. \ ' .\ | | '.'|' ; /|; : | |
| | | | '--'. / ' | '/ :| , ; / / ,. | \ \ ' \ |; : ;' | / || , ; |
| | : \ `--'---' | : / ---' ; : .' \ \ \ |--" | , / | : | ---' |
| | | ,' \ \ .' | , .-./ \ \ | ---`-' \ \ / |
| `----' `---` `--`---' '---" `----' |
+----------------------------------------------------------------------------------------------------+
| You know, for Files! |
| Made from France with Love |
| Source: https://github.com/dadoonet/fscrawler/ |
| Documentation: https://fscrawler.readthedocs.io/ |
`----------------------------------------------------------------------------------------------------'
16:48:34,363 INFO [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [390.9mb/6.8gb=5.57%], RAM [223.2mb/27.4gb=0.8%], Swap [0b/1.9gb=0.0%].
16:48:35,313 INFO [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
16:48:35,314 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
16:48:37,604 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
16:48:37,665 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
16:48:37,672 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
16:48:37,674 INFO [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.15.0
16:48:38,270 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
16:48:38,276 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
16:48:38,280 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
16:48:38,282 INFO [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.15.0
16:48:38,296 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
16:48:38,431 WARN [o.e.c.RestClient] request [PUT http://127.0.0.1:9200/novels?master_timeout=30s&timeout=30s] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
16:48:38,635 WARN [f.p.e.c.f.c.v.ElasticsearchClientV7] failed to create index [novels], disabling crawler...
16:48:38,636 FATAL [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=cluster_block_exception, reason=index [novels] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176) ~[elasticsearch-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2011) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1988) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1745) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1717) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1684) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.IndicesClient.create(IndicesClient.java:134) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.createIndex(ElasticsearchClientV7.java:229) ~[fscrawler-elasticsearch-client-v7-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.createIndex(ElasticsearchClientV7.java:642) ~[fscrawler-elasticsearch-client-v7-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.createIndices(ElasticsearchClientV7.java:474) ~[fscrawler-elasticsearch-client-v7-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerDocumentServiceElasticsearchImpl.createSchema(FsCrawlerDocumentServiceElasticsearchImpl.java:69) ~[fscrawler-core-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:118) ~[fscrawler-core-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:322) [fscrawler-cli-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:298) [fscrawler-cli-2.7.jar:?]
Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://127.0.0.1:9200], URI [/novels?master_timeout=30s&timeout=30s], status line [HTTP/1.1 429 Too Many Requests]
Warnings: [Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security.]
{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [novels] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}],"type":"cluster_block_exception","reason":"index [novels] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"},"status":429}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) ~[elasticsearch-rest-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) ~[elasticsearch-rest-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2082) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1732) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1717) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1684) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at org.elasticsearch.client.IndicesClient.create(IndicesClient.java:134) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.createIndex(ElasticsearchClientV7.java:229) ~[fscrawler-elasticsearch-client-v7-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.createIndex(ElasticsearchClientV7.java:642) ~[fscrawler-elasticsearch-client-v7-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.createIndices(ElasticsearchClientV7.java:474) ~[fscrawler-elasticsearch-client-v7-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerDocumentServiceElasticsearchImpl.createSchema(FsCrawlerDocumentServiceElasticsearchImpl.java:69) ~[fscrawler-core-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:118) ~[fscrawler-core-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:322) [fscrawler-cli-2.7.jar:?]
at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:298) [fscrawler-cli-2.7.jar:?]
16:48:38,668 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler [novels] stopped
16:48:38,674 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler [novels] stopped
Thank you.
I forgot to mention that I'm running Ubuntu 21.04 on a virtual machine on a synology NAS.
Anyway, I doubled the virtual disk space from 100Gb to 200Gb and I no longer get the 'FATAL' . . . . . but I still get 0 docs in my 'novels'
root@volume1:/media/fscrawler# bin/fscrawler /media/fscrawler/novels
01:03:51,058 INFO [f.console] ,----------------------------------------------------------------------------------------------------.
| ,---,. .--.--. ,----.. ,--, 2.7 |
| ,' .' | / / '. / / \ ,--.'| |
| ,---.' || : /`. / | : : __ ,-. .---.| | : __ ,-. |
| | | .'; | |--` . | ;. /,' ,'/ /| /. ./|: : ' ,' ,'/ /| |
| : : : | : ;_ . ; /--` ' | |' | ,--.--. .-'-. ' || ' | ,---. ' | |' | |
| : | |-, \ \ `. ; | ; | | ,'/ \ /___/ \: |' | | / \ | | ,' |
| | : ;/| `----. \| : | ' : / .--. .-. | .-'.. ' ' .| | : / / |' : / |
| | | .' __ \ \ |. | '___ | | ' \__\/: . ./___/ \: '' : |__ . ' / || | ' |
| ' : ' / /`--' /' ; : .'|; : | ," .--.; |. \ ' .\ | | '.'|' ; /|; : | |
| | | | '--'. / ' | '/ :| , ; / / ,. | \ \ ' \ |; : ;' | / || , ; |
| | : \ `--'---' | : / ---' ; : .' \ \ \ |--" | , / | : | ---' |
| | | ,' \ \ .' | , .-./ \ \ | ---`-' \ \ / |
| `----' `---` `--`---' '---" `----' |
+----------------------------------------------------------------------------------------------------+
| You know, for Files! |
| Made from France with Love |
| Source: https://github.com/dadoonet/fscrawler/ |
| Documentation: https://fscrawler.readthedocs.io/ |
`----------------------------------------------------------------------------------------------------'
01:03:51,088 INFO [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [391.9mb/6.8gb=5.58%], RAM [6.9gb/27.4gb=25.52%], Swap [1.9gb/1.9gb=100.0%].
01:03:51,471 INFO [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
01:03:51,471 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
01:03:52,005 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,035 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,041 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,042 INFO [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.15.0
01:03:52,253 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,258 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,263 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,264 INFO [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.15.0
01:03:52,268 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,307 WARN [o.e.c.RestClient] request [PUT http://127.0.0.1:9200/novels?master_timeout=30s&timeout=30s] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,406 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/_cluster/health/novels?master_timeout=30s&level=cluster&timeout=30s&wait_for_status=yellow] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,415 WARN [o.e.c.RestClient] request [PUT http://127.0.0.1:9200/novels_folder?master_timeout=30s&timeout=30s] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,420 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/_cluster/health/novels_folder?master_timeout=30s&level=cluster&timeout=30s&wait_for_status=yellow] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
01:03:52,425 INFO [f.p.e.c.f.FsParserAbstract] FS crawler started for [novels] for [/media/novels] every [1h]
01:03:52,428 WARN [o.e.c.RestClient] request [GET http://127.0.0.1:9200/] returned 1 warnings: [299 Elasticsearch-7.15.0-79d65f6e357953a5b3cbcc5e2c7c21073d89aa29 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."]
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.