Elastic very slow, keep getting timeouts, yet small index


We are evaluating Elastic with the idea to use X-Pack on a self-hosted environment.

The installation of the ELK stack (Elastic, Kibana and Logstash) was successful and we have installed three log gatherer as well:

  • logstash-* : for local system logs on the ELK server
  • metricbeat-* : to receive metrics from one of the dev server
  • winlogbeat-* : to receive windows events from a windows server.

The activity on the three server is very limited, I would say a user is active on each server about 30 mins a day.

The ELK server:

  • Ubuntu Trust 14.04
  • VPS
  • 200 GB of SSD
  • 6 cores, 64bit
  • 8GB of RAM
  • ELK Stack 5.3

From what I have read, this should be way enough for the activity described above, yet I keep getting two errors:

  1. On "winlogbeat-*" > Discover (and other indices), it takes > 30sec and times out.

  2. In the logs in /opt/elk/elasticsearch/elasticsearch.log I have the following message non-stop:

[2017-04-16T16:21:07,794][WARN ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135708] overhead, spent [4.6s] collecting in the last [4.9s]
[2017-04-16T16:21:11,796][INFO ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135712] overhead, spent [352ms] collecting in the last [1s]
[2017-04-16T16:21:18,310][WARN ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135714] overhead, spent [4.9s] collecting in the last [5.5s]
[2017-04-16T16:21:22,311][INFO ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135718] overhead, spent [427ms] collecting in the last [1s]
[2017-04-16T16:21:28,499][WARN ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135720] overhead, spent [4.9s] collecting in the last [5.1s]

a quick htop on the server:

What could be the problem ? We're initially looking for a simple remote-log-viewer of windows event logs and really liked the long-term other features we could use of an ELK stack.


1 Like

I see that your node is in a constant state of garbage collection. I see that you're only running with a 1g heap, which is possibly too small for your data. Yet, can you verify something for me? Can you run the command jps -l -m -v and share the output here? I would like to see all the options that the Elasticsearch JVM was started with, the htop output only shows a truncated version of the command line (also, please share text rather than screenshots). Are you running all of Elasticsearch, Logstash, and Kibana on the same server?

Jason, It seems only the dev version of the sdk has JPS and I don't

root@log-001:/opt/elk-5.3.0-0/java/bin# ls
ControlPanel java javac javaws jcontrol jjs keytool orbd pack200 policytool rmid rmiregistry servertool tnameserv unpack200
root@log-001:/opt/elk-5.3.0-0/java/bin# jps
-bash: jps: command not found

Yes Elasticsearch, Logstash and Kibana are running on the same server

Maybe this can help:

root@log-001:/opt/elk-5.3.0-0/java/bin# ./java -XX:+PrintFlagsFinal -version | grep HeapSize
uintx ErgoHeapSizeLimit = 0 {product}
uintx HeapSizePerGCThread = 87241520 {product}
uintx InitialHeapSize := 132120576 {product}
uintx LargePageHeapSizeThreshold = 134217728 {product}
uintx MaxHeapSize := 2090860544 {product}
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

Thanks, but this is not what I'm looking for (I can already see in your htop output that the heap size is 1g). I need to see the rest of the JVM args on the running Elasticsearch process. Can you install the JDK?

Here you go

root@log-001:/opt/elk-5.3.0-0/java/bin# jps
15959 Jps
4289 Elasticsearch


root@log-001:/opt/elk-5.3.0-0/java/bin# jps -l -m -v
4289 org.elasticsearch.bootstrap.Elasticsearch -d -p/opt/elk-5.3.0-0/elasticsearch/tmp/elasticsearch.pid -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Xms1g -Xmx1g -Des.path.home=/opt/elk-5.3.0-0/elasticsearch
15975 sun.tools.jps.Jps -l -m -v -Dapplication.home=/usr/lib/jvm/java-7-openjdk-amd64 -Xms8m

I suspect your heap is just too small. Can you take a heap dump and share it?

You should change your heap to something higher, start with 2GB, but 4GB would be better.

Also, use our apt repos, it makes life easier :slight_smile:

I ended up re-installing everything using a self-made Ansible script that followed the doc. As you provide an Ansible script for ES, I just had to add kibana and logstash and that was easier than trying to understand what Bitnami packaged in their installer. I have set VM heap size at 4g and it works well so far.

(not sure how to close this issue)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.