CPU High load

Elastic Stack Elasticsearch
1

Hi Team,

I am using ES-1.3.7(version) In sometimes it causes very high load on particular node what can we do for this type of issue.

I Checked with logs too there is no abnormal logs at all.

Please help on this.Thanks in advance.

::: [NODE_NAME][NODE_ID][elasticseasrch][inet[XXX__IP__XXX]]{master=false}

102.2% (510.8ms out of 500ms) cpu usage by thread 'elasticsearch[NODE_NAME][management][T#3]'
4/10 snapshots sharing following 30 elements
sun.reflect.Reflection.getCallerClass(Native Method)
java.lang.Class.getDeclaredFields(Class.java:1805)
org.apache.lucene.util.RamUsageEstimator.shallowSizeOfInstance(RamUsageEstimator.java:382)
org.apache.lucene.util.RamUsageEstimator.shallowSizeOf(RamUsageEstimator.java:360)
org.apache.lucene.util.fst.Outputs.ramBytesUsed(Outputs.java:104)
org.apache.lucene.util.fst.ByteSequenceOutputs.ramBytesUsed(ByteSequenceOutputs.java:152)
org.apache.lucene.util.fst.ByteSequenceOutputs.ramBytesUsed(ByteSequenceOutputs.java:34)
org.apache.lucene.util.fst.FST.ramBytesUsed(FST.java:409)
org.apache.lucene.util.fst.FST.ramBytesUsed(FST.java:427)102.2% (510.8ms out of 500ms) cpu usage by thread 'elasticsearch[NODE_NAME][management][T#3]'
org.apache.lucene.codecs.blocktree.FieldReader.ramBytesUsed(FieldReader.java:187)
org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.ramBytesUsed(BlockTreeTermsReader.java:308)
org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsProducer.ramBytesUsed(BloomFilterPostingsFormat.java:201)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.ramBytesUsed(PerFieldPostingsFormat.java:238)
org.apache.lucene.index.SegmentCoreReaders.ramBytesUsed(SegmentCoreReaders.java:226)
org.apache.lucene.index.SegmentReader.ramBytesUsed(SegmentReader.java:620)
org.elasticsearch.index.engine.internal.InternalEngine.getReaderRamBytesUsed(InternalEngine.java:1152)
org.elasticsearch.index.engine.internal.InternalEngine.segmentsStats(InternalEngine.java:1164)
org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:540)
org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:166)
org.elasticsearch.action.admin.indices.stats.ShardStats.(ShardStats.java:49)
org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:209)
org.elasticsearch.node.service.NodeService.stats(NodeService.java:156)
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:95)
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:277)
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:268)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 11 elements
org.elasticsearch.action.admin.indices.stats.ShardStats.(ShardStats.java:49)
org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:209)
org.elasticsearch.node.service.NodeService.stats(NodeService.java:156)
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:95)
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:277)
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:268)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

2

That is a very old version of Elasticsearch. You should really consider upgrading.

It seems the node is very busy with node management. Do you have a very large number of indices and/or shards in the cluster? Do you have large mappings and/or mappings that are frequently updated dynamically?

3

Thanks.

Yes, we are work with UPGRADATION So, we do reindex to own index.

Indices: 16
Shards: 80

No, we disabled the dynamic mapping already.

How can we fix the node management

4

Can you provide the full output of the cluster stats API to give us a better idea about the state of the cluster?

5

{
"timestamp": 1513328024278,
"cluster_name": "es",
"status": "green",
"indices": {
"count": 16,
"shards": {
"total": 165,
"primaries": 80,
"replication": 1.0625,
"index": {
"shards": {
"min": 10,
"max": 15,
"avg": 10.3125
},
"primaries": {
"min": 5,
"max": 5,
"avg": 5.0
},
"replication": {
"min": 1.0,
"max": 2.0,
"avg": 1.0625
}
}
},
"docs": {
"count": 163556517,
"deleted": 21877877
},
"store": {
"size_in_bytes": 457061600407,
"throttle_time_in_millis": 709241
},
"fielddata": {
"memory_size_in_bytes": 2750871813,
"evictions": 1671603
},
"filter_cache": {
"memory_size_in_bytes": 1811523164,
"evictions": 37689953
},
"id_cache": {
"memory_size_in_bytes": 0
},
"completion": {
"size_in_bytes": 0
},
"segments": {
"count": 2043,
"memory_in_bytes": 9052378464,
"index_writer_memory_in_bytes": 1384276,
"version_map_memory_in_bytes": 680
},
"percolate": {
"total": 1412351,
"time_in_millis": 521584959,
"current": 0,
"memory_size_in_bytes": -1,
"memory_size": "-1b",
"queries": 10623
}
},
"nodes": {
"count": {
"total": 8,
"master_only": 3,
"data_only": 5,
"master_data": 0,
"client": 0
},
"versions": [
"1.3.7"
],
"os": {
"available_processors": 256,
"mem": {
"total_in_bytes": 2163187679232
},
"cpu": [
{
"vendor": "Intel",
"model": "Xeon",
"mhz": 3000,
"total_cores": 32,
"total_sockets": 1,
"cores_per_socket": 32,
"cache_size_in_bytes": 20480,
"count": 8
}
]
},
"process": {
"cpu": {
"percent": 534
},
"open_file_descriptors": {
"min": 846,
"max": 2690,
"avg": 1887
}
},
"jvm": {
"max_uptime_in_millis": 6551538023,
"versions": [
{
"version": "1.7.0_65",
"vm_name": "Java HotSpot(TM) 64-Bit Server VM",
"vm_version": "24.65-b04",
"vm_vendor": "Oracle Corporation",
"count": 8
}
],
"mem": {
"heap_used_in_bytes": 28083885288,
"heap_max_in_bytes": 46170898432
},
"threads": 2985
},
"fs": {
"total_in_bytes": 38981224026112,
"free_in_bytes": 18850588155904,
"available_in_bytes": 17080571420672,
"disk_reads": 187516718,
"disk_writes": 5414807503,
"disk_io_op": 5602324221,
"disk_read_size_in_bytes": 11904572949504,
"disk_write_size_in_bytes": 147740532974080,
"disk_io_size_in_bytes": 159645105923584,
"disk_queue": "1.4",
"disk_service_time": "0.1"
},
"plugins": [
{
"name": "migration",
"version": "NA",
"description": "No description found.",
"url": "/_plugin/migration/",
"jvm": false,
"site": true
},
{
"name": "kopf",
"version": "1.6.1",
"description": "kopf - simple web administration tool for ElasticSearch",
"url": "/_plugin/kopf/",
"jvm": false,
"site": true
},
{
"name": "cloud-aws",
"version": "2.3.0",
"description": "Cloud AWS Plugin",
"jvm": true,
"site": false
},
{
"name": "head",
"version": "NA",
"description": "No description found.",
"url": "/_plugin/head/",
"jvm": false,
"site": true
}
]
}
}

6

Do you have any non-default configuration in your Elasticsearch configuration?

7

we use the default configuration on Elasticsearch.

8

Team, Please help on this.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.