Getting GC error from elasticsearch

[gayathri (Gayathri TR)]

Hi Team,

Kindly suggest any solutions for below errors. Our cluster is in Red state due to this:

2018-10-05T01:30:58,403][WARN ][o.e.m.j.JvmGcMonitorService] [Node_ELK] [gc][young][1301721][238872] duration [2.2s], collections [1]/[2.4s], total [2.2s]/[2.5h], memory [8.8gb]->[8.7gb]/[11.9gb], all_pools {[young] [182.3mb]->[6.3mb]/[266.2mb]}{[survivor] [33.2mb]->[28mb]/[33.2mb]}{[old] [8.6gb]->[8.6gb]/[11.6gb]}

Thanks,
Gayathri T R

[Christian_Dahlqvist]

It looks like you either need to reduce heap usage or scale up/out the cluster to increase the amount of available heap.

[gayathri (Gayathri TR)]

Can you please suggest any method for that. We have already deleted many of the old indices also added below entry in jvm.options file:

-Xms12g
-Xmx12g

Still the cluster is in RED state.

[Christian_Dahlqvist]

What is the full output of the cluster stats API?

[gayathri (Gayathri TR)]

{
"_nodes" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"cluster_name" : "elasticsearch",
"timestamp" : 1539169452721,
"status" : "red",
"indices" : {
"count" : 144,
"shards" : {
"total" : 1308,
"primaries" : 654,
"replication" : 1.0,
"index" : {
"shards" : {
"min" : 2,
"max" : 10,
"avg" : 9.083333333333334
},
"primaries" : {
"min" : 1,
"max" : 5,
"avg" : 4.541666666666667
},
"replication" : {
"min" : 1.0,
"max" : 1.0,
"avg" : 1.0
}
}
},
"docs" : {
"count" : 2166302612,
"deleted" : 89112
},
"store" : {
"size" : "3.8tb",
"size_in_bytes" : 4260628672944,
"throttle_time" : "0s",
"throttle_time_in_millis" : 0
},
"fielddata" : {
"memory_size" : "3mb",
"memory_size_in_bytes" : 3221816,
"evictions" : 0
},
"query_cache" : {
"memory_size" : "164.6mb",
"memory_size_in_bytes" : 172660032,
"total_count" : 432127,
"hit_count" : 276429,
"miss_count" : 155698,
"cache_size" : 3291,
"cache_count" : 7320,
"evictions" : 4029
},
"completion" : {
"size" : "0b",
"size_in_bytes" : 0
},
"segments" : {
"count" : 28697,
"memory" : "12.5gb",
"memory_in_bytes" : 13500790812,
"terms_memory" : "10.8gb",
"terms_memory_in_bytes" : 11634669923,
"stored_fields_memory" : "1.1gb",
"stored_fields_memory_in_bytes" : 1221563432,
"term_vectors_memory" : "0b",
"term_vectors_memory_in_bytes" : 0,
"norms_memory" : "38.8kb",
"norms_memory_in_bytes" : 39744,
"points_memory" : "479.5mb",
"points_memory_in_bytes" : 502887001,
"doc_values_memory" : "135mb",
"doc_values_memory_in_bytes" : 141630712,
"index_writer_memory" : "26.9mb",
"index_writer_memory_in_bytes" : 28298900,
"version_map_memory" : "294.8kb",
"version_map_memory_in_bytes" : 301902,
"fixed_bit_set" : "0b",
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : 1539129606492,
"file_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 5,
"data" : 3,
"coordinating_only" : 0,
"master" : 3,
"ingest" : 5
},
"versions" : [
"5.0.2"
],
"os" : {
"available_processors" : 14,
"allocated_processors" : 14,
"names" : [
{
"name" : "Linux",
"count" : 5
}
],
"mem" : {
"total" : "53.9gb",
"total_in_bytes" : 57920278528,
"free" : "928.1mb",
"free_in_bytes" : 973279232,
"used" : "53gb",
"used_in_bytes" : 56946999296,
"free_percent" : 2,
"used_percent" : 98
}
},
"process" : {
"cpu" : {
"percent" : 45
},
"open_file_descriptors" : {
"min" : 269,
"max" : 1380,
"avg" : 894
}
},
"jvm" : {
"max_uptime" : "20.2d",
"max_uptime_in_millis" : 1747150459,
"versions" : [
{
"version" : "1.8.0_181",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "25.181-b13",
"vm_vendor" : "Oracle Corporation",
"count" : 5
}
],
"mem" : {
"heap_used" : "24gb",
"heap_used_in_bytes" : 25779717504,
"heap_max" : "43.8gb",
"heap_max_in_bytes" : 47122612224
},
"threads" : 291
},
"fs" : {
"total" : "48.1tb",
"total_in_bytes" : 52896775233536,
"free" : "27.6tb",
"free_in_bytes" : 30373739638784,
"available" : "27.6tb",
"available_in_bytes" : 30368552226816,
"spins" : "true"
},
"plugins" : [
{
"name" : "x-pack",
"version" : "5.0.2",
"description" : "Elasticsearch Expanded Pack Plugin",
"classname" : "org.elasticsearch.xpack.XPackPlugin"
}
],
"network_types" : {
"transport_types" : {
"netty4" : 5
},
"http_types" : {
"netty4" : 5
}
}
}
}

Getting below error in Logs:

[2018-10-10T03:50:11,257][INFO ][o.e.m.j.JvmGcMonitorService] [Node1] [gc][1744865] overhead, spent [411ms] collecting in the last [1s]
[2018-10-10T03:50:12,262][INFO ][o.e.m.j.JvmGcMonitorService] [Node1] [gc][1744866] overhead, spent [365ms] collecting in the last [1s]
[2018-10-10T03:50:13,263][INFO ][o.e.m.j.JvmGcMonitorService] [Node1] [gc][1744867] overhead, spent [329ms] collecting in the last [1s]
[2018-10-10T03:50:34,266][INFO ][o.e.m.j.JvmGcMonitorService] [Node1] [gc][1744888] overhead, spent [296ms] collecting in the last [1s]

[gayathri (Gayathri TR)]

Team,

Kindly help us on the above.