Periodic ES ReadTimeout Error

[Ritesh_Agrawal]

I have a 2 node es cluster (4 core 16GB) config. I am getting read timeout error on 10th minute of every hour. All the scheduled processes from the application is stopped. So nothing scheduled from application side. So I would like to know if there is any schedule internal to es that run every hour. I have not changed any time related config in the elasticsearch.yml.

In one of the thread it was mentioned that GC stops all the communication until it finishes. I checked the logs and it shows that application threads are stopped every minute for around ~0.0009 seconds. Does es or jvm run anything periodically?

Can someone provide me few more pointers to debug the issue.
I am using es version 6.5.3

[HenningAndersen]

Hi @Ritesh_Agrawal,

If you are using default GC (CMS), the pauses will normally not affect performance or cause timeouts, since it runs concurrently. If your system is heavily loaded heap wise, GC may struggle though and fall back to a stop the world GC. You should be able to see if this is happening in your GC log files.

Does the timeout happen between a client and Elasticsearch or internally in Elasticsearch? Seeing the full exception trace and other info on the timeout error would be beneficial.

You can have various processes configured that wake up every hour. For instance watcher and rollup could do so if configured.

It could also be worthwhile looking into any other processes running on the host as well as check that the network itself is not at fault.