Hi All,
I am not able to search in Kibana UI for specific string value (which is indexed already) due to "Discover: Bad Gateway" alert pop up issue. But I am able to search for other String values except this case. But we don't have any proxy or firewall issue in our ELK setup. Version we are using is 6.5.4.
But when I start checking ES logs I found below error. I posted along with hot threads stats too. Please advise whether Bad Gateway issue is related to "JvmGcMonitorService" error or it is irrelevant to GC . Thanks for any tips or advises
Elasticsearch logs:
[2019-03-29T04:05:05,350][WARN ][o.e.m.j.JvmGcMonitorService] [prod_portal_node_07_01] [gc][151600] overhead, spent [2.4s] collecting in the last [2.7s]
[2019-03-29T04:05:06,462][INFO ][o.e.m.j.JvmGcMonitorService] [prod_portal_node_07_01] [gc][151601] overhead, spent [486ms] collecting in the last [1.1s]
[2019-03-29T04:05:09,434][WARN ][o.e.m.j.JvmGcMonitorService] [prod_portal_node_07_01] [gc][151602] overhead, spent [2.6s] collecting in the last [2.9s]
[2019-03-29T04:05:10,435][INFO ][o.e.m.j.JvmGcMonitorService] [prod_portal_node_07_01] [gc][151603] overhead, spent [360ms] collecting in the last [1s]
[2019-03-29T04:05:11,495][INFO ][o.e.m.j.JvmGcMonitorService] [prod_portal_node_07_01] [gc][151604] overhead, spent [338ms] collecting in the last [1s]
[2019-03-29T04:05:14,356][WARN ][o.e.m.j.JvmGcMonitorService] [prod_portal_node_07_01] [gc][151605] overhead, spent [2.6s] collecting in the last [2.8s]
[2019-03-29T04:05:15,400][INFO ][o.e.m.j.JvmGcMonitorService] [prod_portal_node_07_01] [gc][151606] overhead, spent [378ms] collecting in the last [1s]
[2019-03-29T04:05:19,916][WARN ][o.e.m.j.JvmGcMonitorService] [prod_portal_node_07_01] [gc][151608] overhead, spent [2.7s] collecting in the last [3.5s]
hot threads:
::: {prod_portal_node_10_01}{2vkrRXZqQ62rEfwif3M4-Q}{l6sJ-wJUSkSCT_ESS21uBA}{host10.corp.xxx.com}{10.192.153.88:9361}{xpack.installed=true}
Hot threads at 2019-03-29T04:14:52.699, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {prod_portal_node_07_01}{elUKdvZHQ4-VvXYlriB8OQ}{B1LQw7u4TD2-j4U9vfX04g}{host07.corp.xxx.com}{10.192.153.86:9361}{xpack.installed=true}
Hot threads at 2019-03-29T04:13:43.561, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {prod_portal_node_09_01}{9FxFi1cqQwGkSvMuWy6BOw}{iqc5mtgrTVCXVtBkrDSvyg}{host09.corp.xxx.com}{10.192.153.87:9361}{xpack.installed=true}
Hot threads at 2019-03-29T04:14:19.769, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {prod_portal_node_08_01}{hq9Uk-SkSLSrA5ZVXhpPWw}{JitYaANFRgK5iqOLmMNYXQ}{host08.corp.xxx.com}{10.192.153.84:9361}{xpack.installed=true}
Hot threads at 2019-03-29T04:14:19.768, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
16.9% (84.5ms out of 500ms) cpu usage by thread 'elasticsearch[prod_portal_node_08_01][[prod-portal-app-error-log-2019.03][2]: Lucene Merge Thread #4602]'
9/10 snapshots sharing following 23 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.lucene.index.MergePolicy$OneMergeProgress.pauseNanos(MergePolicy.java:156)
org.apache.lucene.index.MergeRateLimiter.maybePause(MergeRateLimiter.java:148)
org.apache.lucene.index.MergeRateLimiter.pause(MergeRateLimiter.java:93)
org.apache.lucene.store.RateLimitedIndexOutput.checkRate(RateLimitedIndexOutput.java:78)
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:72)
org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
org.apache.lucene.codecs.lucene50.ForUtil.writeBlock(ForUtil.java:175)
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(Lucene50PostingsWriter.java:286)
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:156)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
unique snapshot
org.apache.lucene.codecs.lucene50.ForUtil.readBlock(ForUtil.java:203)
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockPostingsEnum.refillPositions(Lucene50PostingsReader.java:638)
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockPostingsEnum.nextPosition(Lucene50PostingsReader.java:773)
org.apache.lucene.index.MappingMultiPostingsEnum.nextPosition(MappingMultiPostingsEnum.java:113)
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:145)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
::: {prod_portal_node_06_01}{lzH1jHkFSh6FYg0XQ3zipg}{F3ImROVFT9qMAZ1vMQv4jg}{host06.corp.xxx.com}{10.192.153.85:9361}{xpack.installed=true}
Hot threads at 2019-03-29T04:14:19.766, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: