Failed to process cluster event (put-mapping)

Elastic Stack Elasticsearch
1

Hi, I have below issue from kibana , Any help appreciated.

org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$null$0(ClusterService.java:255) ~[elasticsearch-5.6.0.jar:5.6.0]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_91]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$onTimeout$1(ClusterService.java:254) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.0.jar:5.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]

image
image.png1269Ă—727 40.9 KB

Clus stats
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "elasticsearch",
"timestamp" : 1538136699460,
"status" : "red",
"indices" : {
"count" : 2095,
"shards" : {
"total" : 2095,
"primaries" : 2095,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"primaries" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
},
"docs" : {
"count" : 144264515,
"deleted" : 0
},
"store" : {
"size" : "42.2gb",
"size_in_bytes" : 45368062112,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"fielddata" : {
"memory_size" : "0b",
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"query_cache" : {
"memory_size" : "0b",
"memory_size_in_bytes" : 0,
"total_count" : 0,
"hit_count" : 0,
"miss_count" : 0,
"cache_size" : 0,
"cache_count" : 0,
"evictions" : 0
},
"completion" : {
"size" : "0b",
"size_in_bytes" : 0
},
"segments" : {
"count" : 12670,
"memory" : "389.8mb",
"memory_in_bytes" : 408805910,
"terms_memory" : "308.5mb",
"terms_memory_in_bytes" : 323534475,
"stored_fields_memory" : "20.9mb",
"stored_fields_memory_in_bytes" : 21996216,
"term_vectors_memory" : "0b",
"term_vectors_memory_in_bytes" : 0,
"norms_memory" : "885.8kb",
"norms_memory_in_bytes" : 907072,
"points_memory" : "2.4mb",
"points_memory_in_bytes" : 2575275,
"doc_values_memory" : "57mb",
"doc_values_memory_in_bytes" : 59792872,
"index_writer_memory" : "14.4mb",
"index_writer_memory_in_bytes" : 15190996,
"version_map_memory" : "51.7kb",
"version_map_memory_in_bytes" : 52992,
"fixed_bit_set" : "17.7mb",
"fixed_bit_set_memory_in_bytes" : 18620472,
"max_unsafe_auto_id_timestamp" : 1538121406952,
"file_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 1,
"data" : 1,
"coordinating_only" : 0,
"master" : 1,
"ingest" : 1
},
"versions" : [
"5.6.0"
],
"os" : {
"available_processors" : 4,
"allocated_processors" : 4,
"names" : [
{
"name" : "Windows Server 2012 R2",
"count" : 1
}
],
"mem" : {
"total" : "7.9gb",
"total_in_bytes" : 8589463552,
"free" : "790.3mb",
"free_in_bytes" : 828723200,
"used" : "7.2gb",
"used_in_bytes" : 7760740352,
"free_percent" : 10,
"used_percent" : 90
}
},
"process" : {
"cpu" : {
"percent" : -1
},
"open_file_descriptors" : {
"min" : -1,
"max" : -1,
"avg" : 0
}
},
"jvm" : {
"max_uptime" : "36.1m",
"max_uptime_in_millis" : 2167865,
"versions" : [
{
"version" : "1.8.0_91",
"vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
"vm_version" : "25.91-b14",
"vm_vendor" : "Oracle Corporation",
"count" : 1
}
],
"mem" : {
"heap_used" : "3.7gb",
"heap_used_in_bytes" : 4030366968,
"heap_max" : "3.9gb",
"heap_max_in_bytes" : 4260102144
},
"threads" : 50
},
"fs" : {
"total" : "1022.8gb",
"total_in_bytes" : 1098301566976,
"free" : "883.8gb",
"free_in_bytes" : 949041754112,
"available" : "883.8gb",
"available_in_bytes" : 949041754112
},
"plugins" : ,
"network_types" : {
"transport_types" : {
"netty4" : 1
},
"http_types" : {
"netty4" : 1
}
}
}
}

2

You probably have too many shards per node.

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

Here 2095 shards for 40gb is a way too much.
You probably need something like 2 to 4 shards at most...

That's a waste of resources.

1 Like
3

Thanks for the reply,

I will re work on this configuration However if I extend 30 GB more does the current issue solve?
Please this is effecting my prod system.

4

I don't get it. If you extend what?

Here as I said, you have too many shards per node.
Also running in "production" with one single node is not really safe.

Anyway, you should reduce the number of shards.
If you are using time based data, may be remove the unneeded indices (old ones).

One last solution could be to start more nodes if you really wish to keep all those shards around.

Think about a Shard as a single database instance. Would you run 2000 databases instances on a single machine with 8gb of RAM?

5

I have cleared cache for now and got data back in pace.

I clearly understand you point now.

At the moment I am using time based data. I am going to work on this to remove old indices.

Thanks for the quick reply.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.