Tasks are stuck for hours for index creation & setting update

Elastic Stack Elasticsearch
1

Our cluster has 4k+ indices, 36 data nodes(8c & 32g). ,3 mn (4c & 16g)
Write is on 10k+ & read is very less
No of primary shards - 4k+, replica 1
Es version: 5.x
All tasks are running for hrs.
Logs throwing RemoteTransportException[[axcdfer][__IP__][__PATH__]]; nested: ProcessClusterEventTimeoutException[failed to process cluster event (update-settings) within 30s];

cluster stats response

{
    "_nodes":
    {
        "total": 39,
        "successful": 39,
        "failed": 0
    },
    "status": "green",
    "indices":
    {
        "count": 4205,
        "shards":
        {
            "total": 8472,
            "primaries": 4232,
            "replication": 1.001890359168242,
            "index":
            {
                "shards":
                {
                    "min": 2,
                    "max": 8,
                    "avg": 2.01474435196195
                },
                "primaries":
                {
                    "min": 1,
                    "max": 4,
                    "avg": 1.0064209274673008
                },
                "replication":
                {
                    "min": 1.0,
                    "max": 2.0,
                    "avg": 1.0019024970273485
                }
            }
        },
        "docs":
        {
            "count": 687645501,
            "deleted": 227615845
        },
        "store":
        {
            "size_in_bytes": 461674571016,
            "throttle_time_in_millis": 0
        },
        "fielddata":
        {
            "memory_size_in_bytes": 0,
            "evictions": 0
        },
        "query_cache":
        {
            "memory_size_in_bytes": 52614032,
            "total_count": 5769365,
            "hit_count": 12711,
            "miss_count": 5756654,
            "cache_size": 1036,
            "cache_count": 1649,
            "evictions": 613
        },
        "completion":
        {
            "size_in_bytes": 1065914
        },
        "segments":
        {
            "count": 39606,
            "memory_in_bytes": 3019471272,
            "terms_memory_in_bytes": 2792779786,
            "stored_fields_memory_in_bytes": 75863936,
            "term_vectors_memory_in_bytes": 0,
            "norms_memory_in_bytes": 29512832,
            "points_memory_in_bytes": 7033838,
            "doc_values_memory_in_bytes": 114280880,
            "index_writer_memory_in_bytes": 2649482309,
            "version_map_memory_in_bytes": 814845713,
            "fixed_bit_set_memory_in_bytes": 0,
            "max_unsafe_auto_id_timestamp": 1677385060118,
            "file_sizes":
            {}
        }
    },
    "nodes":
    {
        "count":
        {
            "total": 39,
            "data": 36,
            "coordinating_only": 0,
            "master": 3,
            "ingest": 36
        },
        "versions":
        [
            "5.4"
        ],
        "os":
        {
            "available_processors": 300,
            "allocated_processors": 300,
            "names":
            [
                {
                    "count": 39
                }
            ],
            "mem":
            {
                "total_in_bytes": 1237275987968,
                "free_in_bytes": 116169551872,
                "used_in_bytes": 1121106436096,
                "free_percent": 9,
                "used_percent": 91
            }
        },
        "process":
        {
            "cpu":
            {
                "percent": 104
            },
            "open_file_descriptors":
            {
                "min": 1768,
                "max": 2384,
                "avg": 2294
            }
        },
        "jvm":
        {
            "max_uptime_in_millis": 163535887,
            "mem":
            {
                "heap_used_in_bytes": 252877825880,
                "heap_max_in_bytes": 648072658944
            },
            "threads": 7062
        },
        "fs":
        {
            "total_in_bytes": 3667610210304,
            "free_in_bytes": 3196143022080,
            "available_in_bytes": 3009946152960
        },
        "network_types":
        {
            "transport_types":
            {
                "netty4": 39
            },
            "http_types":
            {
                "filter-jetty": 39
            }
        }
    }
}

image
image1838×514 133 KB

2

This is a very old version which has been EOL for a very long time. It sounds like you are suffering from slow cluster state updates, which could be due to the large number of shards in the cluster or nodes being overloaded.

I noticed that you mentioned that this cluster is an AWS managed cluster before you edited it away (you should have left it in there as it is a critical piece of information). This makes it very hard to troubleshoot as AWS as far as I know use custom plugins and maybe even their own version.

If it is indeed managed by AWS I would recommend reaching out to their support.

I would recommend upgrading to the latest version as a lot of improvements have been made with respect to how the cluster state is managed and how clusters scale.

3

Thanks, @Christian_Dahlqvist. An immediate upgrade is not possible, too many clients are involved. So we've to scale the cluster, can you suggest, how many more data nodes we've to add?

image
image696×988 41.9 KB

4

Adding more data nodes is not going to help if cluster updates are slow. It is likely to instead make the issue worse as the cluster state need to be propagated to more nodes.

I would in general recommend the following:

  • You have a lot of small shards which is inefficient. There is information about each index and shard in the cluster state, e.g. locations, settings and mappings, so reducing the number of shards could help.
  • Try to reduce the number of cluster state updates. Shard relocations and shard creations result in cluster state updates. If you are using dynamic mapping, each new field added also results in a cluster state update.
  • Ensure nodes are not suffering from long GC (can delay propagation of state) and that none of the nodes are overloaded or have saturated IO. Check the logs for this.
5

Thanks, @Christian_Dahlqvist. Will check these.

6

@Christian_Dahlqvist, Went through your recommended points & also this post Now the issue boils sdown to more number of shards & long GC. So we can't reduce shards immediately so If we keep the node's memory below 75% by vertical scaling, do we have a chance of resolving this issue?

7

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

8

If you are using Opensearch I can not really answer. It started out as a fork of Elasticsearch but has changed significantly since and there are third party plugins that could impact memory usage and performance.

Scaling vertically is given that you are running such an ancient version not really possible. That version does not support G1GC and running a very large heap with CMS can cause a lot longer GC. I do not see any easy way to addressing this issue by scaling nodes or the cluster. In my opinion you need to either upgrade and/or address the underlying issue, which is large number of shars and frequent cluster state changes.

9

Got it, Thanks @Christian_Dahlqvist.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.