Elasticsearch stops working, says failed to update shard information

Elastic Stack Elasticsearch
1

I am running elasticsearch 6.8.1 on centos 7. Out of a sudden, elasticsearch service stopped working giving the following error in the elasticsearch.log file. I restarted the service then it is working normally. But how do I avoid this error in future? Is there any settings to turn off this auto update? Thanks in advance.

[2019-07-03T02:11:32,687][WARN ][o.e.c.InternalClusterInfoService] [ZqPNVKY] Failed to update shard information for ClusterInfoUpdateJob within 15s timeout
[2019-07-03T02:11:32,684][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ZqPNVKY] fatal error in thread [elasticsearch[ZqPNVKY][search][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.util.ArrayUtil.growExact(ArrayUtil.java:302) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
2

You seem to have run out of heap space, so either reduce or optimise how much you use or add more heap to the cluster. If you provide more details on your use case and setup someone might be able to assist, but now there is not much to go on.

3

Thanks @Christian_Dahlqvist How can we avoid this automatic update process from happening?

4

How many indices and shards do you have in the cluster? What is the size and hardware specification of the cluster?

5

Here is the information -

GET _cluster/health
{
  "cluster_name" : "my-first-cluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2291,
  "active_shards" : 2291,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3181,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 41.86769005847953
}

The cluster is sitting on a centos server with 2 vCPU 8GB RAM and 50GB hard drive.
Thanks in advance.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.