Kibana "Error in visualization: Request times out after 30000ms"

[Kok_How_Teh]

I am using 7.3.1 and I keep getting "Error in visualizatio: Request times out after 30000ms" which is a major blocker to make it into production use. I have increased Elasticsearch JVM Options to "-Xms5g -Xmx5g" but to no avail. This is the Java process of the Elasticsearch running in kubernetes cluster:

elastic+     1     0  1 Sep24 ?        00:18:53 /usr/share/elasticsearch/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-12037710348029652064 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Xms5g -Xmx5g -Djna.tmpdir=/tmp -Dio.netty.allocator.type=pooled -XX:MaxDirectMemorySize=2684354560 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch

# /usr/share/elasticsearch/jdk/bin/java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
     intx CompilerThreadStackSize                  = 1024                                   {pd product} {default}
   size_t ErgoHeapSizeLimit                        = 0                                         {product} {default}
   size_t HeapSizePerGCThread                      = 43620760                                  {product} {default}
   size_t InitialHeapSize                          = 255852544                                 {product} {ergonomic}
   size_t LargePageHeapSizeThreshold               = 134217728                                 {product} {default}
   size_t MaxHeapSize                              = 4085252096                                {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 5825164                                {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 122916538                              {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 122916538                              {pd product} {ergonomic}
     intx ThreadStackSize                          = 1024                                   {pd product} {default}
     intx VMThreadStackSize                        = 1024                                   {pd product} {default}
openjdk version "12.0.2" 2019-07-16
OpenJDK Runtime Environment (build 12.0.2+10)
OpenJDK 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)

[flash1293 (Joe Reuter)]

Hi @Kok_How_Teh,

Do you see any errors in the Elasticsearch logs? Is this just happening for a single specific visualization or for each visualization even though it's not querying a lot of data?

[Kok_How_Teh]

It happens in a dashboard which contains 4 or 5 visualizations like Pie, Timelion, Bar and Coordinate Map.

[flash1293 (Joe Reuter)]

Could you try to make out which visualization is causing the timeout by disabling them one by one? If you found the problematic one, could you send the request this visualization is actually sending out (https://www.elastic.co/guide/en/kibana/7.3/vis-inspector.html) along with an export of the configuration of that visualization?

[Kok_How_Teh]

Alright. I will do this tomorrow morning back in the office. Thanks.

[Kok_How_Teh]

@flash1293, I see this error log from Elasticsearch:

[elasticsearch-0 elasticsearch] "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:305) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", 
[elasticsearch-0 elasticsearch] "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", 
[elasticsearch-0 elasticsearch] "at java.lang.Thread.run(Thread.java:835) [?:?]"] } 
[elasticsearch-0 elasticsearch] {"type": "server", "timestamp": "2019-09-26T02:37:02,951+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-0", "cluster.uuid": "tF0YHrUaSuO1Kjuo5OKHXA", "node.id": "mTr1xrfiSB-jhng32cehfQ",  "message": "All shards failed for phase: [query]"  } 
[elasticsearch-0 elasticsearch] {"type": "server", "timestamp": "2019-09-26T02:37:02,952+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-0", "cluster.uuid": "tF0YHrUaSuO1Kjuo5OKHXA", "node.id": "mTr1xrfiSB-jhng32cehfQ",  "message": "path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager}" ,  
[elasticsearch-0 elasticsearch] "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:305) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]", 
[elasticsearch-0 elasticsearch] "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", 
[elasticsearch-0 elasticsearch] "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", 
[elasticsearch-0 elasticsearch] "at java.lang.Thread.run(Thread.java:835) [?:?]"] } 
[elasticsearch-0 elasticsearch] {"type": "deprecation", "timestamp": "2019-09-26T02:37:05,185+0000", "level": "WARN", "component": "o.e.d.s.a.b.h.DateHistogramAggregationBuilder", "cluster.name": "elasticsearch", "node.name": "elasticsearch-0", "cluster.uuid": "tF0YHrUaSuO1Kjuo5OKHXA", "node.id": "mTr1xrfiSB-jhng32cehfQ",  "message": "[interval] on [date_histogram] is deprecated, use [fixed_interval] or [calendar_interval] in the future."  } 

[Kok_How_Teh]

[flash1293 (Joe Reuter)]

Can you limit it to a single visualization causing the problems and then paste the request from the inspector (see https://www.elastic.co/guide/en/kibana/7.3/vis-inspector.html) here?

[Kok_How_Teh]

[flash1293 (Joe Reuter)]

@Kok_How_Teh, that's just the error message when the timeout occurs. Can you export and paste the saved object of the visualization here along with the actual request that times out? You can see the request in the chrome developer tools or in the inspector panel above when the timeout doesn't occur.