CPU running high only on one node at time in the ES cluster

sbk-elk · February 28, 2021, 1:33am

Not sure why only one node in the cluster has such high CPU when there is nothing happening in the cluster.

top - 00:34:43 up 137 days, 17:43,  1 user,  load average: 1.24, 2.14, 3.15
Tasks: 198 total,   1 running, 197 sleeping,   0 stopped,   0 zombie
%Cpu(s): 27.7 us,  1.9 sy,  0.0 ni, 68.6 id,  1.7 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 61676952 total,  1712800 free, 34528200 used, 25435952 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 26000248 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8302 elastic+  20   0  297.6g  38.9g   6.9g S **476.1** 66.1 252422:37 java
  567 root      20   0       0      0      0 S   1.3  0.0 971:45.39 jbd2/dm-0-8
  106 root      20   0       0      0      0 S   1.0  0.0 191:39.65 kswapd0
    9 root      20   0       0      0      0 S   0.3  0.0 329:15.73 rcu_sched
 1024 root      20   0  121220  15916   3264 S   0.3  0.0 364:46.79 google_osconfig
 3390 root      20   0  160044   2312   1536 R   0.3  0.0   0:00.32 top
    1 root      20   0  191164   3124   1652 S   0.0  0.0   1:24.79 systemd
    2 root      20   0       0      0      0 S   0.0  0.0  10:48.88 kthreadd
    4 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S   0.0  0.0  21:36.47 ksoftirqd/0

warkolm · February 28, 2021, 11:33pm

What version are you on?
What is in the logs?
What about hot threads?

sbk-elk · March 1, 2021, 2:37am

Version -- 7.6.2

Logs:
Is there a specific log you are looking for or any stack trace that you expect that may cause the CPU spike because I don't see any logs that point to any issues or are there any errors. Sometimes i see "failed to execute bulk item (index)" for a few indexes but that doesnt create any issues in the cluster or nodes(as per my understanding).

Is there way i can upload the hot threads out put, it is a big log I some how cant attach it here.

Thanks for your response @warkolm

warkolm · March 1, 2021, 2:54am

Any/all logs can be relevant, even if you aren't spotting anything.

Use gist/pastebin/etc.

sbk-elk · March 1, 2021, 5:02am

gist.github.com

https://gist.github.com/sushanth45/7df48d27b35be2baba212d9986504b6b

hothreads.txt


::: {  node-1}{8WjFXK39T9ykS5HOO22A3Q}{NCWvcpSOT5-AHpPWyQMwng}{  x.x.x.217}{  x.x.x.217:9300}{di}{xpack.installed=true}
   Hot threads at 2021-03-01T02:27:36.545Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {  node-2}{NAOT4Kg_Tj6SYONld6epgA}{mby8TiBfSEK_ozy4yvOsVQ}{  x.x.x.216}{  x.x.x.216:9300}{di}{xpack.installed=true}
   Hot threads at 2021-03-01T02:27:36.546Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   68.1% (340.2ms out of 500ms) cpu usage by thread 'elasticsearch[  node-2][[index-2021.03.01][0]: Lucene Merge Thread #82]'
     10/10 snapshots sharing following 9 elements
       app//org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)

This file has been truncated. show original

logs

[2021-03-01T00:00:23,210][DEBUG][o.e.a.b.TransportShardBulkAction] [node-2] [index-ability-hz-2021.03.01][0] failed to execute bulk item (index) index {[index-ability-hz-2021.03.01][_doc][0uwV63cBclN6oM9f0bDv], source[{"@timestamp":"2021-03-01T00:00:11.755Z","name":"caching_location_tombstone_counter","type":"counter","host":"eastavailability-hz-6","count":0.0}]}
java.lang.IllegalArgumentException: mapper [count] cannot be changed from type [long] to [float]
	at org.elasticsearch.index.mapper.MappedFieldType.checkTypeName(MappedFieldType.java:154) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.index.mapper.MappedFieldType.checkCompatibility(MappedFieldType.java:167) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.index.mapper.MapperMergeValidator.validateFieldMapper(MapperMergeValidator.java:93) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.index.mapper.MapperMergeValidator.validateNewMappers(MapperMergeValidator.java:65) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:483) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:415) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:315) ~[elasticsearch-7.6.2.jar:7.6.2]

This file has been truncated. show original

sbk-elk · March 1, 2021, 5:03am

@warkolm I have uploaded the cluster details as well in the comments fyi. Please let me know if you need any further details that can help further check on it.

system · March 29, 2021, 5:04am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.