FSCrawler : Retry connection, when it gets failure

Elastic Stack Elasticsearch
1

Hi David,
Please share your suggestions. We are using fscrawler-2.4..

When we startup.. getting below connection error.. is there way to look (or) wait for connection success, then continue the process within the _settings.json script.

05:15:08,480 DEBUG [f.p.e.c.f.FsCrawler] Starting job [processformstemplates]...
05:15:08,629 TRACE [f.p.e.c.f.FsCrawler] settings used for this crawler: [{
"name" : "processformstemplates0",
"fs" : {
"url" : "/usr/share/fscrawler/data",
"update_rate" : "15m",
"excludes" : [ "/~" ],
"json_support" : false,
"filename_as_id" : true,
"add_filesize" : true,
"remove_deleted" : false,
"add_as_inner_object" : false,
"store_source" : false,
"index_content" : true,
"attributes_support" : false,
"raw_metadata" : false,
"xml_support" : false,
"index_folders" : true,
"lang_detect" : false,
"continue_on_error" : false,
"pdf_ocr" : true,
"ocr" : {
"language" : "eng"
}
},
"Elasticsearch" : {
"nodes" : [ {
"host" : "Elasticsearch-documents-http.app-doc.svc.cluster.local",
"port" : 9200,
"scheme" : "HTTP"
} ],
"index" : "templatesindex",
"bulk_size" : 100,
"flush_interval" : "5s",
"username" : "elastic"
},
"rest" : {
"port" : 0
}
}]
05:15:09,159 WARN [f.p.e.c.f.c.ElasticsearchClientManager] failed to create Elasticsearch client, disabling crawler...
05:15:09,159 FATAL [f.p.e.c.f.FsCrawler] Fatal error received while running the crawler: [Connection refused]
05:15:09,160 DEBUG [f.p.e.c.f.FsCrawler] error caught
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_121]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_121]
at org.Elasticsearch.client.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171) ~[Elasticsearch-rest-client-6.0.0-beta1.jar:6.0.0-beta1]
at org.Elasticsearch.client.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145) ~[Elasticsearch-rest-client-6.0.0-beta1.jar:6.0.0-beta1]
at org.Elasticsearch.client.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) ~[Elasticsearch-rest-client-6.0.0-beta1.jar:6.0.0-beta1]
at org.Elasticsearch.client.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192) ~[Elasticsearch-rest-client-6.0.0-beta1.jar:6.0.0-beta1]
at org.Elasticsearch.client.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[Elasticsearch-rest-client-6.0.0-beta1.jar:6.0.0-beta1]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_121]

Thank You,
Joseph

2

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

Now back to the question. What is your Elasticsearch version?

What gives:

curl Elasticsearch-documents-http.app-doc.svc.cluster.local:9200
3

Hi David,
Sorry, i update the format.. in between.. i am placing this..

image
image1266×386 21.4 KB

Our Elastic Version is 6.8.14.

4

Could you upgrade to FSCrawler 2.7?

5

Yes David, even we thought of doing before. But unfortunately, our other related applications all are in JDK 1.8, If we move to FS Crawler 2.7, then our ELK version needs to be changed and JDK version also. Thanks

6

FSCrawler 2.7 has a distribution for es6.
I don't understand why you would have to upgrade Elasticsearch

7

For this combination, in linux, we have faced an issue. So, we did with FS Crawler 2.6.

8

What kind of issue?

9

that we had 6 months back, then we continue with FSCrawler 2.4 and ELK 6.8.14. Yes.

10

I'm not following.
You should upgrade and fix the issue you faced 6 months ago. Not sure what it was and if I can help on this.

11

Hi David, We are trying to upgrade to FS Crawler 2.7 Version.
Thanks.

12

FYI, I will build in the next days hopefully a version 2.9 of FSCrawler.

13

Sure, Thanks David. We will look on it..
Mean time, we have upgraded to FS Crawler 2.7 and we got the same error, what i have mentioned above 'Connection Refused error'.
Could You please suggest.

And we have referred below thread and tried to follow:

Thanks a lot for your time.

14

In Additionally, we tried for curl command..

curl Elasticsearch-documents-http.app-doc.svc.cluster.local:9200
curl: (6) Could not resolve host: Elasticsearch-documents-http.app-doc.svc.cluster.local; Unknown error

15

That means that your Elasticsearch cluster can not be reached from the machine where FSCrawler is running.

What are the Elasticsearch logs when it starts?

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image764×92 16.4 KB

16

Yes.. Thanks a lot David.. As you mentioned same kind of issue.. due to proxy request did not receive and our team got fixed this.

1 Like
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.