Getting Timeouts from elasticsearch

Elastic Stack Elasticsearch
1

Hi Team,

All of our Nodes are Up and Green in state. But we are getting below timeout errors in log and all indeces and search actions are too slow in Kibana.

We are using ELK Version 5.0.2.

[2017-09-24T22:35:05,709][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [Node_1_v5] failed to execute on node [CSg3P5mVQZ2pcVvtwbKjLA]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [Node_2__v5][localhost:9301][cluster:monitor/nodes/stats[n]] request_id [7718005] timed out after [15001ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:842) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.0.2.jar:5.0.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
[2017-09-24T22:35:05,772][WARN ][o.e.a.a.c.n.s.TransportNodesStatsAction] [Node_1_v5] not accumulating exceptions, excluding exception from response
org.elasticsearch.action.FailedNodeException: Failed node [CSg3P5mVQZ2pcVvtwbKjLA]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:247) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$300(TransportNodesAction.java:160) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:219) [elasticsearch-5.0.2.jar:5.0.2]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:957) [elasticsearch-5.0.2.jar:5.0.2]

Thanks,

Gayathri

2

How much data do you have in the cluster? What is the specification of your Elasticsearch nodes (CPU, RAM, heap, type of storage)? How many indices and shards do you have in the cluster? How many indices and shards are you querying through Kibana when you are experiencing problems?

Do you have monitoring installed so you can identify what is limiting cluster performance?

3

Hi Christian,

We have almost 1 year of data on the cluster.
CPU:4GB
RAM : 16 GB
Heap : 12 GB
Type of Storage: NAS : 16T
No.of Indeces: 480
Shards: 3
Replicas: 1
We are getting error while querying 1 month of data on Kibana.

4

How much data do you have in the cluster? You should also ideally set heap to 50% of RAM and the file system cache is essential for Elasticsearch performance.

Please also read this blog post about shards and sharding.

I would also recommend monitoring disk I/O, iowait and CPU usage while you are querying in order to see what is limiting performance. As you are using NAS storage, it could very well be that slow storage is a contributing factor.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.