Hi,
Running ELK 6.4 with x-pack activated (trial licence) on a single Redhat server.
First sign that something was going wrong was Kibana not letting me log in, I got to the logon screen but my password stopped working. I decided to restart the kibana service. From there I found that it was no longer listening on 5601 (connection refused), checked and re-checked my config but still couldn't figure it out...went down a rabbit hole thinking my issue was kibana.
Actually, after some time I realised that elastic wasn't working!
I've pasted my error log below from var/logs/elasticsearch
my jvm.options settings are default.
Xms represents the initial size of total heap space
Xmx represents the maximum size of total heap space
-Xms1g
-Xmx1g
For some reason I can't find the error file: ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log
Java HotSpot(TM) 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 7 2018 00:56:38 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 1878724k(353792k free), swap 16777212k(16771316k free)
CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=75 -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:GCLogFileSize=67108864 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=174485504 -XX:MaxTenuringThreshold=6 -XX:NewSize=174485504 -XX:NumberOfGCLogFiles=32 -XX:OldPLABSize=16 -XX:OldSize=348971008 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC
2018-10-21T06:26:18.873+0000: 2.932: Total time for which application threads were stopped: 0.0015610 seconds, Stopping threads took: 0.0000324 seconds
2018-10-21T06:26:19.877+0000: 3.937: Total time for which application threads were stopped: 0.0090252 seconds, Stopping threads took: 0.0000482 seconds
2018-10-21T06:26:20.885+0000: 4.944: Total time for which application threads were stopped: 0.0000958 seconds, Stopping threads took: 0.0000384 seconds
2018-10-21T06:26:22.137+0000: 6.197: Total time for which application threads were stopped: 0.0001617 seconds, Stopping threads took: 0.0000549 seconds
2018-10-21T06:26:23.217+0000: 7.276: Total time for which application threads were stopped: 0.0000991 seconds, Stopping threads took: 0.0000313 seconds
2018-10-21T06:26:23.373+0000: 7.433: Total time for which application threads were stopped: 0.0001242 seconds, Stopping threads took: 0.0000329 seconds
2018-10-21T06:26:23.575+0000: 7.634: Total time for which application threads were stopped: 0.0001425 seconds, Stopping threads took: 0.0000358 seconds
2018-10-21T06:26:24.575+0000: 8.634: Total time for which application threads were stopped: 0.0000868 seconds, Stopping threads took: 0.0000248 seconds
2018-10-21T06:26:24.748+0000: 8.807: Total time for which application threads were stopped: 0.0001554 seconds, Stopping threads took: 0.0000312 seconds
2018-10-21T06:26:25.301+0000: 9.360: Total time for which application threads were stopped: 0.0001851 seconds, Stopping threads took: 0.0000329 seconds
Heap
par new generation total 153344K, used 62738K [0x00000000c0000000, 0x00000000ca660000, 0x00000000ca660000)
eden space 136320K, 46% used [0x00000000c0000000, 0x00000000c3d44ab0, 0x00000000c8520000)
from space 17024K, 0% used [0x00000000c8520000, 0x00000000c8520000, 0x00000000c95c0000)
to space 17024K, 0% used [0x00000000c95c0000, 0x00000000c95c0000, 0x00000000ca660000)
concurrent mark-sweep generation total 878208K, used 0K [0x00000000ca660000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 13099K, capacity 13488K, committed 13696K, reserved 1060864K
class space used 1702K, capacity 1893K, committed 1920K, reserved 1048576K
-XX:HeapDumpPath=/var/lib/elasticsearch
-rw------- 1 elasticsearch elasticsearch 1345458873 Oct 6 21:11 java_pid22570.h prof
-rw------- 1 elasticsearch elasticsearch 1340236957 Oct 6 21:36 java_pid16598.h
Any help appreciated