Fatal Error in thread Elasticsearch

Elastic Stack Elasticsearch
1

Hi Team,

We have 7 node elasticsearch cluster we are facing sometimes facing the same issue in the following topic.

Please find our cluster statistics please let us know are we doing something wrong with shard size or Heap size.

{
"_nodes" : {
"total" : 7,
"successful" : 7,
"failed" : 0
},
"cluster_name" : "cul-prim-prod",
"cluster_uuid" : "X4QaFpncQeyyb_hxC-G54Q",
"timestamp" : 1578555450588,
"status" : "green",
"indices" : {
"count" : 46,
"shards" : {
"total" : 292,
"primaries" : 146,
"replication" : 1.0,
"index" : {
"shards" : {
"min" : 2,
"max" : 10,
"avg" : 6.3478260869565215
},
"primaries" : {
"min" : 1,
"max" : 5,
"avg" : 3.1739130434782608
},
"replication" : {
"min" : 1.0,
"max" : 1.0,
"avg" : 1.0
}
}
},
"docs" : {
"count" : 3362009859,
"deleted" : 25461
},
"store" : {
"size_in_bytes" : 10665225023735
},
"fielddata" : {
"memory_size_in_bytes" : 567400,
"evictions" : 0
},
"query_cache" : {
"memory_size_in_bytes" : 970304418,
"total_count" : 6764188043,
"hit_count" : 1083259,
"miss_count" : 6763104784,
"cache_size" : 1353748,
"cache_count" : 1374718,
"evictions" : 20970
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 5305,
"memory_in_bytes" : 12608131101,
"terms_memory_in_bytes" : 9957812822,
"stored_fields_memory_in_bytes" : 2158594512,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 55541504,
"points_memory_in_bytes" : 379701971,
"doc_values_memory_in_bytes" : 56480292,
"index_writer_memory_in_bytes" : 1494975972,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 910208,
"max_unsafe_auto_id_timestamp" : 1578554123176,
"file_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 7,
"data" : 7,
"coordinating_only" : 0,
"master" : 5,
"ingest" : 5
},
"versions" : [
"6.7.1"
],
"os" : {
"available_processors" : 56,
"allocated_processors" : 56,
"names" : [
{
"name" : "Linux",
"count" : 7
}
],
"pretty_names" : [
{
"pretty_name" : "Ubuntu 16.04.6 LTS",
"count" : 7
}
],
"mem" : {
"total_in_bytes" : 472925831168,
"free_in_bytes" : 17409150976,
"used_in_bytes" : 455516680192,
"free_percent" : 4,
"used_percent" : 96
}
},
"process" : {
"cpu" : {
"percent" : 174
},
"open_file_descriptors" : {
"min" : 614,
"max" : 780,
"avg" : 694
}
},
"jvm" : {
"max_uptime_in_millis" : 10798483409,
"versions" : [
{
"version" : "1.8.0_232",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "25.232-b09",
"vm_vendor" : "Private Build",
"count" : 5
},
{
"version" : "1.8.0_222",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "25.222-b10",
"vm_vendor" : "Private Build",
"count" : 2
}
],
"mem" : {
"heap_used_in_bytes" : 112234882040,
"heap_max_in_bytes" : 240030056448
},
"threads" : 742
},
"fs" : {
"total_in_bytes" : 37701522087936,
"free_in_bytes" : 26274215526400,
"available_in_bytes" : 24374004559872
},
"plugins" : ,
"network_types" : {
"transport_types" : {
"security4" : 7
},
"http_types" : {
"security4" : 7
}
}
}
}

2

this seems to be an exception, that you are running out of memory (and thus the process gets killed instead of running in a ghost state).

Are there any more logs before the event happening?

Have you configured the same heap size for all of your systems?

Did you change any of your elasticsearch configuration? Are you running big aggregations or other operations that may potentially require a lot of memory?

Also, is this a customized JDK build?

3

Thanks for support.We ran queries with aggregations post that issue occured.Do you have any suggestions for running queries or aggregations.Please let us know.

Regards,
Vivek

4

Can we configure more than 32 GB Heap (Say 48 GB ) with 64 GB RAM.?

Regards,
Vivek

5

Don't ask the same question multiple times please. Elasticsearch Heapsize query is a duplicate then.

6

Sure...Thought of maintaining different thread for this different question.

Thanks,
Vivek

7

It seems your heap size is set quite close to 32gb which may mean that you are not using compressed pointers. The recommendation is generally to not set it larger that around 30GB. You may want to try decreasing it to this and see if it make a difference.

It may also help if you describe the use case and the types of aggregations you are running as someone might be able to help you improve this.

8

Thank you @Christian_Dahlqvist..We will try that and get back to you ...

Regards,
Vivek

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.