Get error 'no such index [pdf_test]' after use fscrawer to create index

Elastic Stack Elasticsearch
1

FSCrawler verssion: 2.7

when I first run the command: bin\fscrawler --config_dir ./DS pdf_test,
it showed info:

job [pdf_test] does not exist
Do you want to create it (Y/N)?
Y
Settings have been created in [.\DS\pdf_test\_settings.yaml]. Please review and edit before relaunch.

And it created dir DS and pdf_test. and then I changed the settings.yaml.
after I run the bin\fscrawler --config_dir ./DS pdf_test --loop 1,

I got nothing on the cmd windows.
there is no info or warning showing on the cmd windows.

and I search on the kibana, there is no index pdf_test exist. how can I troubleshooting the issue?

2

Could you check the logs dir?

4

the log fscrawer.log showing:
17:48:20,045 INFO [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [109.3mb/1.9gb=5.43%], RAM [2.1gb/7.8gb=26.81%], Swap [1.4gb/14.2gb=10.4%].
17:48:20,246 INFO [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
17:48:20,761 WARN [f.p.e.c.f.c.v.ElasticsearchClientV7] failed to create elasticsearch client on Elasticsearch{nodes=[http://10.22.1.201:9200], index='pdf_test', indexFolder='pdf_test_folder', bulkSize=100, flushInterval=5s, byteSize=10mb, username='null', pipeline='null', pathPrefix='null', sslVerification='true'}, disabling crawler...
17:48:20,762 FATAL [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=missing authentication credentials for REST request [/]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1900) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1877) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1634) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1606) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1573) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:774) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.getVersion(ElasticsearchClientV7.java:179) ~[fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.checkVersion(ElasticsearchClient.java:192) ~[fscrawler-elasticsearch-client-base-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.start(ElasticsearchClientV7.java:152) ~[fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerManagementServiceElasticsearchImpl.start(FsCrawlerManagementServiceElasticsearchImpl.java:60) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:116) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:307) [fscrawler-cli-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:283) [fscrawler-cli-2.7-SNAPSHOT.jar:?]
Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://10.22.1.201:9200], URI [/], status line [HTTP/1.1 401 Unauthorized]
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) ~[elasticsearch-rest-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) ~[elasticsearch-rest-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1621) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1606) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1573) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:774) ~[elasticsearch-rest-high-level-client-7.11.1.jar:7.11.1]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.getVersion(ElasticsearchClientV7.java:179) ~[fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.checkVersion(ElasticsearchClient.java:192) ~[fscrawler-elasticsearch-client-base-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.start(ElasticsearchClientV7.java:152) ~[fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerManagementServiceElasticsearchImpl.start(FsCrawlerManagementServiceElasticsearchImpl.java:60) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:116) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:307) [fscrawler-cli-2.7-SNAPSHOT.jar:?]
at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:283) [fscrawler-cli-2.7-SNAPSHOT.jar:?]
17:48:20,768 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler [pdf_test] stopped
17:48:20,769 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler [pdf_test] stopped

5

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

6

The problem seems to be a login/password not set:

7

where can I set the login/passward?

8

Here: Elasticsearch settings — FSCrawler 2.7-SNAPSHOT documentation

9

Thanks a lot. it's resolved after following the steps.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.