Shards keep re-initializing

Elastic Stack Elasticsearch
1

I've got a 7 node cluster, each node has 64GB of RAM, 2x XEON E5-2623v4 CPUs(16 Threads) and 12TB of SSD in RAID6. It's hosting 56.63TB of data across 187 Indices and 2,346 shards. Elasticsearch v6.8.3 behind Graylog.

Today it's been exhibiting some very odd behavior. I had a node restart, not unusual, but initializing the shards has been very slow. More than once, I've seen an index initial all of its shards only to see a few minutes later that it is trying to initialize 1 or more shards again.

There is not an unusual amount of data being written to the cluster, though the logs have been showing the following error on multiple nodes:
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [30066013583/28gb], which is larger than the limit of [30064771072/28gb], usages [request=304/304b, fielddata=13521194527/12.5gb, in_flight_requests=17003986/16.2mb, accounting=16527814766/15.3gb]

2

What is the output of the cluster stats API? What does your node configuration look like?

3

The problem resolved itself after I restarted Elasticsearch on all of my nodes. They had gotten themselves into a death spiral of garbage collection, which doesn't happen frequently, but more often than I'd prefer. It seems to happen more often as my nodes approach 75-80% disk capacity, which I assume means too many shards per node.
Here are the output of the API calls.
/_cluster/stats

{
  "_nodes": {
    "total": 8,
    "successful": 8,
    "failed": 0
  },
  "cluster_name": "sac0",
  "cluster_uuid": "x8ubl0h8TJ2HOWMS4gVQrw",
  "timestamp": 1590518501572,
  "status": "green",
  "indices": {
    "count": 189,
    "shards": {
      "total": 2374,
      "primaries": 1187,
      "replication": 1,
      "index": {
        "shards": {
          "min": 2,
          "max": 14,
          "avg": 12.56084656084656
        },
        "primaries": {
          "min": 1,
          "max": 7,
          "avg": 6.28042328042328
        },
        "replication": {
          "min": 1,
          "max": 1,
          "avg": 1
        }
      }
    },
    "docs": {
      "count": 37842616058,
      "deleted": 214693
    },
    "store": {
      "size_in_bytes": 62044674450928
    },
    "fielddata": {
      "memory_size_in_bytes": 28182739128,
      "evictions": 0
    },
    "query_cache": {
      "memory_size_in_bytes": 3540755629,
      "total_count": 22209116,
      "hit_count": 6308974,
      "miss_count": 15900142,
      "cache_size": 119539,
      "cache_count": 189819,
      "evictions": 70280
    },
    "completion": {
      "size_in_bytes": 0
    },
    "segments": {
      "count": 4398,
      "memory_in_bytes": 115582144034,
      "terms_memory_in_bytes": 84918220380,
      "stored_fields_memory_in_bytes": 27911216744,
      "term_vectors_memory_in_bytes": 0,
      "norms_memory_in_bytes": 510592,
      "points_memory_in_bytes": 2697780350,
      "doc_values_memory_in_bytes": 54415968,
      "index_writer_memory_in_bytes": 259602368,
      "version_map_memory_in_bytes": 2312680,
      "fixed_bit_set_memory_in_bytes": 3263080,
      "max_unsafe_auto_id_timestamp": 1590451202602,
      "file_sizes": {

      }
    }
  },
  "nodes": {
    "count": {
      "total": 8,
      "data": 7,
      "coordinating_only": 1,
      "master": 7,
      "ingest": 7
    },
    "versions": [
      "6.8.3"
    ],
    "os": {
      "available_processors": 128,
      "allocated_processors": 128,
      "names": [
        {
          "name": "Linux",
          "count": 8
        }
      ],
      "pretty_names": [
        {
          "pretty_name": "Debian GNU/Linux 9 (stretch)",
          "count": 8
        }
      ],
      "mem": {
        "total_in_bytes": 539593818112,
        "free_in_bytes": 16193302528,
        "used_in_bytes": 523400515584,
        "free_percent": 3,
        "used_percent": 97
      }
    },
    "process": {
      "cpu": {
        "percent": 94
      },
      "open_file_descriptors": {
        "min": 677,
        "max": 2799,
        "avg": 2341
      }
    },
    "jvm": {
      "max_uptime_in_millis": 331555667,
      "versions": [
        {
          "version": "1.8.0_202",
          "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
          "vm_version": "25.202-b08",
          "vm_vendor": "Oracle Corporation",
          "count": 2
        },
        {
          "version": "1.8.0_212",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "vm_version": "25.212-b03",
          "vm_vendor": "AdoptOpenJDK",
          "count": 1
        },
        {
          "version": "1.8.0_152",
          "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
          "vm_version": "25.152-b16",
          "vm_vendor": "Oracle Corporation",
          "count": 5
        }
      ],
      "mem": {
        "heap_used_in_bytes": 230268751440,
        "heap_max_in_bytes": 301721452544
      },
      "threads": 1496
    },
    "fs": {
      "total_in_bytes": 86327228289024,
      "free_in_bytes": 24143018700800,
      "available_in_bytes": 23231637172224
    },
    "plugins": [
      {
        "name": "readonlyrest",
        "version": "1.18.5",
        "elasticsearch_version": "6.8.3",
        "java_version": "1.8",
        "description": "Safely expose Elasticsearch REST API",
        "classname": "tech.beshu.ror.es.ReadonlyRestPlugin",
        "extended_plugins": [

        ],
        "has_native_controller": false
      }
    ],
    "network_types": {
      "transport_types": {
        "netty4": 8
      },
      "http_types": {
        "ssl_netty4": 8
      }
    }
  }
}
4

/_nodes/jd1QMwhlQUCdJEFN4FO2ng

{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "sac0",
  "nodes": {
    "jd1QMwhlQUCdJEFN4FO2ng": {
      "name": "elasticsearch-sac0-0004",
      "transport_address": "10.2.0.182:9300",
      "host": "10.2.0.182",
      "ip": "10.2.0.182",
      "version": "6.8.3",
      "build_flavor": "default",
      "build_type": "deb",
      "build_hash": "0c48c0e",
      "total_indexing_buffer": 4294967296,
      "roles": [
        "master",
        "data",
        "ingest"
      ],
      "attributes": {
        "ml.machine_memory": "67449200640",
        "ml.max_open_jobs": "20",
        "xpack.installed": "true",
        "ml.enabled": "true"
      },
      "settings": {
        "pidfile": "/var/run/elasticsearch/elasticsearch.pid",
        "cluster": {
          "name": "sac0"
        },
        "node": {
          "attr": {
            "xpack": {
              "installed": "true"
            },
            "ml": {
              "machine_memory": "67449200640",
              "max_open_jobs": "20",
              "enabled": "true"
            }
          },
          "name": "elasticsearch-sac0-0004"
        },
        "path": {
          "data": [
            "/var/lib/elasticsearch"
          ],
          "logs": "/var/log/elasticsearch",
          "home": "/usr/share/elasticsearch"
        },
        "discovery": {
          "zen": {
            "ping": {
              "unicast": {
                "hosts": [
                  "elasticsearch-sac0-0000.backblaze.com",
                  "elasticsearch-sac0-0001.backblaze.com",
                  "elasticsearch-sac0-0002.backblaze.com"
                ]
              }
            }
          }
        },
        "client": {
          "type": "node"
        },
        "http": {
          "type": "ssl_netty4",
          "type.default": "netty4"
        },
        "transport": {
          "type": {
            "default": "netty4"
          },
          "features": {
            "x-pack": "true"
          }
        },
        "xpack": {
          "security": {
            "enabled": "false"
          }
        },
        "network": {
          "host": "0.0.0.0"
        }
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Linux",
        "pretty_name": "Debian GNU/Linux 9 (stretch)",
        "arch": "amd64",
        "version": "4.9.0-7-amd64",
        "available_processors": 16,
        "allocated_processors": 16
      },
      "process": {
        "refresh_interval_in_millis": 1000,
        "id": 1429,
        "mlockall": false
      },
      "jvm": {
        "pid": 1429,
        "version": "1.8.0_152",
        "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
        "vm_version": "25.152-b16",
        "vm_vendor": "Oracle Corporation",
        "start_time_in_millis": 1590186973460,
        "mem": {
          "heap_init_in_bytes": 42949672960,
          "heap_max_in_bytes": 42949672960,
          "non_heap_init_in_bytes": 2555904,
          "non_heap_max_in_bytes": 0,
          "direct_max_in_bytes": 42949672960
        },
        "gc_collectors": [
          "G1 Young Generation",
          "G1 Old Generation"
        ],
        "memory_pools": [
          "Code Cache",
          "Metaspace",
          "G1 Eden Space",
          "G1 Survivor Space",
          "G1 Old Gen"
        ],
        "using_compressed_ordinary_object_pointers": "false",
        "input_arguments": [
          "-Xms40g",
          "-Xmx40g",
          "-XX:+UseG1GC",
          "-XX:+AlwaysPreTouch",
          "-Xss1m",
          "-Djava.awt.headless=true",
          "-Dfile.encoding=UTF-8",
          "-Djna.nosys=true",
          "-Djdk.io.permissionsUseCanonicalPath=true",
          "-Dio.netty.noUnsafe=true",
          "-Dio.netty.noKeySetOptimization=true",
          "-Dio.netty.recycler.maxCapacityPerThread=0",
          "-Dlog4j.shutdownHookEnabled=false",
          "-Dlog4j2.disable.jmx=true",
          "-Dlog4j.skipJansi=true",
          "-XX:+HeapDumpOnOutOfMemoryError",
          "-XX:+PrintGCDetails",
          "-XX:+PrintGCTimeStamps",
          "-XX:+PrintGCDateStamps",
          "-XX:+PrintClassHistogram",
          "-XX:+PrintTenuringDistribution",
          "-XX:+PrintGCApplicationStoppedTime",
          "-Xloggc:/var/log/elasticsearch/gc.log",
          "-XX:+UseGCLogFileRotation",
          "-XX:NumberOfGCLogFiles=32",
          "-XX:GCLogFileSize=128M",
          "-Dcom.sun.management.jmxremote",
          "-Dcom.sun.management.jmxremote.port=9100",
          "-Dcom.sun.management.jmxremote.authenticate=false",
          "-Dcom.sun.management.jmxremote.ssl=false",
          "-Des.path.home=/usr/share/elasticsearch",
          "-Des.path.conf=/etc/elasticsearch",
          "-Des.distribution.flavor=default",
          "-Des.distribution.type=deb"
        ]
      },
      "thread_pool": {
        "watcher": {
          "type": "fixed",
          "min": 50,
          "max": 50,
          "queue_size": 1000
        },
        "force_merge": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "ml_datafeed": {
          "type": "fixed",
          "min": 20,
          "max": 20,
          "queue_size": 200
        },
        "fetch_shard_started": {
          "type": "scaling",
          "min": 1,
          "max": 32,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "listener": {
          "type": "fixed",
          "min": 8,
          "max": 8,
          "queue_size": -1
        },
        "ml_autodetect": {
          "type": "fixed",
          "min": 80,
          "max": 80,
          "queue_size": 80
        },
        "index": {
          "type": "fixed",
          "min": 16,
          "max": 16,
          "queue_size": 200
        },
        "refresh": {
          "type": "scaling",
          "min": 1,
          "max": 8,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "generic": {
          "type": "scaling",
          "min": 4,
          "max": 128,
          "keep_alive": "30s",
          "queue_size": -1
        },
        "rollup_indexing": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 4
        },
        "warmer": {
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "search": {
          "type": "fixed_auto_queue_size",
          "min": 25,
          "max": 25,
          "queue_size": 1000
        },
        "ccr": {
          "type": "fixed",
          "min": 32,
          "max": 32,
          "queue_size": 100
        },
        "flush": {
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "fetch_shard_store": {
          "type": "scaling",
          "min": 1,
          "max": 32,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "management": {
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "ml_utility": {
          "type": "fixed",
          "min": 80,
          "max": 80,
          "queue_size": 500
        },
        "get": {
          "type": "fixed",
          "min": 16,
          "max": 16,
          "queue_size": 1000
        },
        "analyze": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": 16
        },
        "write": {
          "type": "fixed",
          "min": 16,
          "max": 16,
          "queue_size": 200
        },
        "snapshot": {
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "search_throttled": {
          "type": "fixed_auto_queue_size",
          "min": 1,
          "max": 1,
          "queue_size": 100
        }
      },
      "transport": {
        "bound_address": [
          "[::]:9300"
        ],
        "publish_address": "10.2.0.182:9300",
        "profiles": {

        }
      },
      "http": {
        "bound_address": [
          "[::]:9200"
        ],
        "publish_address": "10.2.0.182:9200",
        "max_content_length_in_bytes": 104857600
      },
      "plugins": [
        {
          "name": "readonlyrest",
          "version": "1.18.5",
          "elasticsearch_version": "6.8.3",
          "java_version": "1.8",
          "description": "Safely expose Elasticsearch REST API",
          "classname": "tech.beshu.ror.es.ReadonlyRestPlugin",
          "extended_plugins": [

          ],
          "has_native_controller": false
        }
      ],
     
      "ingest": {
        "processors": [
          {
            "type": "append"
          },
          {
            "type": "bytes"
          },
          {
            "type": "convert"
          },
          {
            "type": "date"
          },
          {
            "type": "date_index_name"
          },
          {
            "type": "dissect"
          },
          {
            "type": "dot_expander"
          },
          {
            "type": "drop"
          },
          {
            "type": "fail"
          },
          {
            "type": "foreach"
          },
          {
            "type": "geoip"
          },
          {
            "type": "grok"
          },
          {
            "type": "gsub"
          },
          {
            "type": "join"
          },
          {
            "type": "json"
          },
          {
            "type": "kv"
          },
          {
            "type": "lowercase"
          },
          {
            "type": "pipeline"
          },
          {
            "type": "remove"
          },
          {
            "type": "rename"
          },
          {
            "type": "script"
          },
          {
            "type": "set"
          },
          {
            "type": "set_security_user"
          },
          {
            "type": "sort"
          },
          {
            "type": "split"
          },
          {
            "type": "trim"
          },
          {
            "type": "uppercase"
          },
          {
            "type": "urldecode"
          },
          {
            "type": "user_agent"
          }
        ]
      }
    }
  }
}
5

What does your node configuration (elasticsearch.yml) look like? Do you have any non-default setings that could be causing problems? I wonder if you may not have minimum_master_nodes set correctly (should be 4 with 7 master eligible nodes) so what you are seeing could be a side effect of a split-brain occurring?

There are also a few things I have noted:

  • You are running ES 6.8 with Java8 and G1GC, which is not recommended nor supported.
  • You seem to have an unusually large heap configured.
  • You are using a third-party plugin which may interfere with the default distribution and is an unknown (at least to me).
6

I do not have anything special configured in elasticsearch.yml.

Which part of the ES 6.8/Java8/G1GC combination isn't recommended? I'm stuck with 6.8 because Graylog does not support ES7 yet. There's a note in the 6.8 documentation that says there were issues with G1GC before 8u40, but I assumed 1.8.0_152 was new enough to not have any issues.

The 40G heap was a part of some experimenting, it wasn't any worse than 30G. My servers have 64GB of RAM, so I was trying to make the most of what's available.

An upgrade to Java 11is rolling out within my company, so that is definitely a bit I can twiddle.

Thanks so much for the advice.

7

Do you have minimum_master_nodes correctly set?

For information about JVM and products I would recommend the support matrix. This blog post is also quite useful.

I believe G1GC is only supported on Java11 onwards.

8

I converted our staging cluster to Java 11 and the number of garbage collection events is way down.
I'm working on setting the minimum_master_nodes now.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.