Elastic very slow, keep getting timeouts, yet small index

[Pablo_Albrecht]

Hi,

We are evaluating Elastic with the idea to use X-Pack on a self-hosted environment.

The installation of the ELK stack (Elastic, Kibana and Logstash) was successful and we have installed three log gatherer as well:

• logstash-* : for local system logs on the ELK server
• metricbeat-* : to receive metrics from one of the dev server
• winlogbeat-* : to receive windows events from a windows server.

The activity on the three server is very limited, I would say a user is active on each server about 30 mins a day.

The ELK server:

• Ubuntu Trust 14.04
• VPS
• 200 GB of SSD
• 6 cores, 64bit
• 8GB of RAM
• ELK Stack 5.3

From what I have read, this should be way enough for the activity described above, yet I keep getting two errors:

1. On "winlogbeat-*" > Discover (and other indices), it takes > 30sec and times out.

2. In the logs in /opt/elk/elasticsearch/elasticsearch.log I have the following message non-stop:

[2017-04-16T16:21:07,794][WARN ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135708] overhead, spent [4.6s] collecting in the last [4.9s]
[2017-04-16T16:21:11,796][INFO ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135712] overhead, spent [352ms] collecting in the last [1s]
[2017-04-16T16:21:18,310][WARN ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135714] overhead, spent [4.9s] collecting in the last [5.5s]
[2017-04-16T16:21:22,311][INFO ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135718] overhead, spent [427ms] collecting in the last [1s]
[2017-04-16T16:21:28,499][WARN ][o.e.m.j.JvmGcMonitorService] [SFm5d9i] [gc][135720] overhead, spent [4.9s] collecting in the last [5.1s]

a quick htop on the server:

What could be the problem ? We're initially looking for a simple remote-log-viewer of windows event logs and really liked the long-term other features we could use of an ELK stack.

Regards

[jasontedor]

I see that your node is in a constant state of garbage collection. I see that you're only running with a 1g heap, which is possibly too small for your data. Yet, can you verify something for me? Can you run the command jps -l -m -v and share the output here? I would like to see all the options that the Elasticsearch JVM was started with, the htop output only shows a truncated version of the command line (also, please share text rather than screenshots). Are you running all of Elasticsearch, Logstash, and Kibana on the same server?

[Pablo_Albrecht]

Jason, It seems only the dev version of the sdk has JPS and I don't

root@log-001:/opt/elk-5.3.0-0/java/bin# ls
ControlPanel java javac javaws jcontrol jjs keytool orbd pack200 policytool rmid rmiregistry servertool tnameserv unpack200
root@log-001:/opt/elk-5.3.0-0/java/bin# jps
-bash: jps: command not found
root@log-001:/opt/elk-5.3.0-0/java/bin#

Yes Elasticsearch, Logstash and Kibana are running on the same server

Maybe this can help:

root@log-001:/opt/elk-5.3.0-0/java/bin# ./java -XX:+PrintFlagsFinal -version | grep HeapSize
uintx ErgoHeapSizeLimit = 0 {product}
uintx HeapSizePerGCThread = 87241520 {product}
uintx InitialHeapSize := 132120576 {product}
uintx LargePageHeapSizeThreshold = 134217728 {product}
uintx MaxHeapSize := 2090860544 {product}
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

[jasontedor]

Thanks, but this is not what I'm looking for (I can already see in your htop output that the heap size is 1g). I need to see the rest of the JVM args on the running Elasticsearch process. Can you install the JDK?

[Pablo_Albrecht]

Here you go

root@log-001:/opt/elk-5.3.0-0/java/bin# jps
15959 Jps
4289 Elasticsearch
root@log-001:/opt/elk-5.3.0-0/java/bin#

and

root@log-001:/opt/elk-5.3.0-0/java/bin# jps -l -m -v
4289 org.elasticsearch.bootstrap.Elasticsearch -d -p/opt/elk-5.3.0-0/elasticsearch/tmp/elasticsearch.pid -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Xms1g -Xmx1g -Des.path.home=/opt/elk-5.3.0-0/elasticsearch
15975 sun.tools.jps.Jps -l -m -v -Dapplication.home=/usr/lib/jvm/java-7-openjdk-amd64 -Xms8m
root@log-001:/opt/elk-5.3.0-0/java/bin#

[jasontedor]

I suspect your heap is just too small. Can you take a heap dump and share it?

[warkolm (Mark Walkom)]

You should change your heap to something higher, start with 2GB, but 4GB would be better.

Also, use our apt repos, it makes life easier

[Pablo_Albrecht]

I ended up re-installing everything using a self-made Ansible script that followed the doc. As you provide an Ansible script for ES, I just had to add kibana and logstash and that was easier than trying to understand what Bitnami packaged in their installer. I have set VM heap size at 4g and it works well so far.

(not sure how to close this issue)