Hello,
My Elasticsearch stops to index when after certain number of logs are sent to elastic via logstash and I keep seeing
[2018-03-08T15:44:34,677][INFO ][o.e.m.j.JvmGcMonitorService] [aTs5egi] [gc][14522] overhead, spent [460ms] collecting in the last [1s]
2018-03-08T15:44:35.562-0800 ERROR logstash/async.go:235 Failed to publish events caused by: read tcp 10.244.9.182:30789->10.244.9.182:5044: i/o timeout
2018-03-08T15:44:35.562-0800 ERROR logstash/async.go:235 Failed to publish events caused by: read tcp 10.244.9.182:30789->10.244.9.182:5044: i/o timeout
2018-03-08T15:44:35.643-0800 ERROR logstash/async.go:235 Failed to publish events caused by: client is not connected
[2018-03-08T15:44:35,677][WARN ][o.e.m.j.JvmGcMonitorService] [aTs5egi] [gc][14523] overhead, spent [563ms] collecting in the last [1s]
2018-03-08T15:44:36.643-0800 ERROR pipeline/output.go:92 Failed to publish events: client is not connected
[2018-03-08T15:44:36,678][INFO ][o.e.m.j.JvmGcMonitorService] [aTs5egi] [gc][14524] overhead, spent [411ms] collecting in the last [1s]
2018-03-08T15:44:37.443-0800 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":980,"time":986},"total":{"ticks":13470,"time":13480,"value":13470},"user":{"ticks":12490,"time":12494}},"info":{"ephemeral_id":"56497923-61d1-4f43-8caa-896a9f2f8fab","uptime":{"ms":1500016}},"memstats":{"gc_next":17106224,"memory_alloc":14692256,"memory_total":736280280}},"filebeat":{"harvester":{"open_files":99,"running":98}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":9,"failed":18432,"total":18432},"read":{"errors":3},"write":{"bytes":911900}},"pipeline":{"clients":1,"events":{"active":4117,"retry":24576}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":4.91,"15":4.75,"5":5.09,"norm":{"1":2.455,"15":2.375,"5":2.545}}}}}}
Below is the result of jps -l -m -v:
-bash-4.1$ jps -l -m -v
21862 org.logstash.Logstash -f app-store-pipeline.conf -Xms10g -Xmx10g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom
7608 org.elasticsearch.bootstrap.Elasticsearch -Xms10g -Xmx10g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.18wEpBH6 -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/scratch/elk/elasticsearch-6.2.2 -Des.path.conf=/scratch/elk/elasticsearch-6.2.2/config
23258 sun.tools.jps.Jps -l -m -v -Dapplication.home=/ds1/home/oracle/jdk/jdk1.8.0_102 -Xms8m
The total count stops at 4096:
{
"count" : 4096,
"_shards" : {
"total" : 440,
"successful" : 440,
"skipped" : 0,
"failed" : 0
}
}
It has been a hour since my last log is indexed into elasticsearch. Could someone please help? Thanks!