Hi everyone,
We're experiencing some slowness on our elasticsearch 6.2.3 clusters when performing update_mapping and create indices tasks. One such cluster consists of 3 master nodes (4 cores / 14gb memory / 7gb heap), 3 client nodes (same setup), 40 hot data nodes (8 cores / 64gb memory / 30.5gb heap and 1.4tb local ssd disks) and 8 warm data nodes (8 cores / 32gb memory / 16gb heap and 5tb spinning disks). This cluster contains ~14,000 primary shards (25,000 total active shards) spread across ~8,900 indices, and calling GET _cluster/state shows the cluster state weights ~94mb. This is the current response to GET _cluster/health:
{
"cluster_name": "tier-1-01",
"status": "green",
"timed_out": false,
"number_of_nodes": 54,
"number_of_data_nodes": 48,
"active_primary_shards": 13722,
"active_shards": 25204,
"relocating_shards": 8,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100
}
As we're indexing our customer's data, where we cannot control its (changing) structure, we're heavily reliant on dynamic mapping. Also, to prevent backlogs during a UTC rollover event, we're opening new indices in advance in a daily manner, where around 1,200 new create index tasks are run during low-activity. Each index matches an index template, which injects our custom analyzers, the dynamic mapping, some custom fields and 3 aliases per index (for index rollover, searching etc). These indices live to the retention period of each customer (between 1 and 30 days, mostly 1 or 14). We also have a relatively high variation of docs count / size_in_byte between indices: some contain none (230b single shard indices where the user did not eventually send data that day), while some have much larger amounts of data (e.g. 1tb spread across 30-40 shards).
As discussed on github #30370, we're experiencing many timeouts on update_mapping when running our background tasks to open indices for the next day. This graph shows the high correlation between the two tasks:
It seems that create index tasks (taking between 4.5 and 5.5 seconds each), which elasticsearch assigns 'urgent' priority, put aside update_mapping tasks which are assigned 'high' priority, greatly increasing the chance of these tasks to complete after their 30 seconds timeout. However, as @DavidTurner and @bleskes suggest, this may be caused by some misconfiguration or another issue which we'd like some help investigating.
Let me know if any further information is needed, and we'll dive deeper accordingly.
Thanks!
