Elasticsearch timeout issues

Elastic Stack Elasticsearch
1

Hi,

I am repeatedly facing timeout issues while visualizing data ki Kibana.

Here are the logs I saw when the timeout used to occur:

[2018-05-31T09:05:08,785][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [GXT24Jx] failed to execute on node [GXT24JxFR2u09ohBu50mBg]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [GXT24Jx][10.30.197.90:9300][cluster:monitor/nodes/stats[n]] request_id [16296211] timed out after [15602ms]
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:982) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
[2018-05-31T09:05:11,684][WARN ][o.e.m.j.JvmGcMonitorService] [GXT24Jx] [gc][417220] overhead, spent [2.7s] collecting in the last [2.9s]
[2018-05-31T09:05:11,714][ERROR][o.e.x.w.t.s.ExecutableScriptTransform] [GXT24Jx] failed to execute [script] transform for [MS4pzhrVTKa3P7BrRUfwGA_elasticsearch_cluster_status_62e53d8c-2691-4d55-856f-a41271e5c2a4-2018-05-31T09:05:08.779Z]org.elasticsearch.script.ScriptException: runtime error
        at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:101) ~[?:?]
        at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ctx.vars.is_new = ctx.vars.fails_check && !ctx.vars.not_resolved;ctx.vars.is_resolve ...:1070) ~[?:?]
        at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:105) ~[?:?]
        at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.doExecute(ExecutableScriptTransform.java:69) ~[x-pack-watcher-6.2.4.jar:6.2.4]
        at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:53) ~[x-pack-watcher-6.2.4.jar:6.2.4]
        at org.elasticsearch.xpack.watcher.transform.script.ExecutableScriptTransform.execute(ExecutableScriptTransform.java:38) ~[x-pack-watcher-6.2.4.jar:6.2.4]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:476) ~[x-pack-watcher-6.2.4.jar:6.2.4]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:317) ~[x-pack-watcher-6.2.4.jar:6.2.4]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$6(ExecutionService.java:421) ~[x-pack-watcher-6.2.4.jar:6.2.4]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:575) [x-pack-watcher-6.2.4.jar:6.2.4]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.rangeCheck(ArrayList.java:657) ~[?:1.8.0_151]
        at java.util.ArrayList.get(ArrayList.java:433) ~[?:1.8.0_151]
        at org.elasticsearch.painless.PainlessScript$Script.execute(ctx.vars.email_recipient = (ctx.payload.kibana_settings.hits.total > 0) ? ctx.payload.kibana_settings.hits.hits[0]._source.kibana_settings.xpack.default_admin_email : null;ctx.vars.is_new = ctx.vars.fails_check && !ctx.vars.not_resolved;ctx.vars.is_resolve ...:347) ~[?:?]
        ... 12 more

Here are the specs of ELK stack.
No. of ES nodes = 1
Amount to data ingested daily - few GBs only.
CPU cores = 4
CPU utilization, iowait and memory utilization are normal when the timeout occurs.

2

What version of the Elastic Stack?
What the heap size?
What is the viz query that is being ran, how many indices does it touch?

3

Elastic stack version - 6.2.4
Heap size - 1G
No. of indices it touches - It varies depending on visualization. But it gets timed out even when viewing the indices from settings in Kibana.

I guess heap size might need to be increased.

4

Please do start with the increase in heap size and then visualise in Kibana and feedback any issues remaining.

5

Looks like that was the issue. Changed heap size to 4G and everything is working just fine.

Thanks for the pointers.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.