CPU load avg goes very high all of a sudden

Elastic Stack Elasticsearch
1

Hi am using elastic search 2.4 one node deployment

pumping huge constant traffic, its stable for almost one day and suddenly started very high load.

machines are of 14cpu, and ~30gb allocated to lucen + es, when i do hot threads i get below

::: {metrics-master-aad1b7c303407c7335949cc692983dee}{pOWqx6Y-RpGZwkR3jdXU5g}{10.117.113.5}{10.117.113.5:9300}{max_local_storage_nodes=1, data=false, master=true}
  Hot threads at 2018-06-19T04:28:38.888Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {metrics-datastore_9fb54563e76338973c973647200f4646}{i9o3HBAPR2qcFn5I4XvwgA}{10.117.113.15}{10.117.113.15:9300}{max_local_storage_nodes=1, master=false}
 Hot threads at 2018-06-19T04:28:38.837Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

 100.5% (502.6ms out of 500ms) cpu usage by thread 'elasticsearch[metrics-datastore_9fb54563e76338973c973647200f4646][[39c42350-7f3e-4aaf-9db2-7cd1db53fabe842804860-2018.06.18.23-logs][0]: Lucene Merge Thread #1657]'
 2/10 snapshots sharing following 14 elements
   org.apache.lucene.codecs.blocktree.SegmentTermsEnum.next(SegmentTermsEnum.java:962)
   org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:275)
   org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:301)
   org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:195)
   org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:438)
  org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
   org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
   org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
   org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
   org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
   org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   




 org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
 3/10 snapshots sharing following 13 elements
   org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:275)
   org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:301)
   org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:195)
   org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:438)
   org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
   org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
   org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
   org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
   org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
   org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
 2/10 snapshots sharing following 13 elements
   org.apache.lucene.index.MappedMultiFields$MappedMultiTermsEnum.postings(MappedMultiFields.java:127)
   org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
   org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:1041)
   org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:456)
   org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
   org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
   org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
   org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
   org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
   org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)
   org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
 3/10 snapshots sharing following 10 elements
   org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:456)
   org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
   org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
   org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
   org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
   org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
   org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)
   org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.