Memory spikes on ILM transition

Elastic Stack Elasticsearch
1

We use hot-warm-cold architecture

policy

{
  "filebeat": {
    "version": 39128,
    "modified_date": "2023-12-11T04:05:20.404Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "2h",
              "max_primary_shard_size": "50gb"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "warm": {
          "min_age": "1h",
          "actions": {
            "allocate": {
              "number_of_replicas": 1,
              "include": {},
              "exclude": {},
              "require": {}
            },
            "readonly": {},
            "set_priority": {
              "priority": 50
            }
          }
        },
        "cold": {
          "min_age": "1d",
          "actions": {
            "allocate": {
              "number_of_replicas": 1,
              "include": {},
              "exclude": {},
              "require": {}
            },
            "readonly": {},
            "set_priority": {
              "priority": 0
            }
          }
        },
        "delete": {
          "min_age": "11d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      }
    }
  }
}

in "hot" initially we also write "number_of_replicas": 1 for index
and as we don't use any transform actions like "forcemerge" or "shrink" we expect that it's just network copy from "hot" nodes to "warm" nodes, but according metrics we see spikes in network and spikes in memory (off heap) which is problem for us as it always triggers memory alerts >90% usage.

Can someone advice what do to to prevent such memory spikes?
our hot node setup

resources:
        limits:
          cpu: '8'
          memory: 48Gi
        requests:
          cpu: '6'
          memory: 48Gi

Xms/Xmx/MaxDirectMemorySize default auto values generated by Elasticsearch itself

And maybe some general ILM good practices, maybe we need change something in our policy?

2

What is the full output of the cluster stats API?

Why are you rolling over every 2 hours? How many primary and replica shards does each index have during indexing? How much data are you ingesting per day?

3

cluster stats

{
  "_nodes": {
    "total": 19,
    "successful": 19,
    "failed": 0
  },
  "cluster_name": "elastic-search",
  "cluster_uuid": "xxxxxx",
  "timestamp": 1702280147268,
  "status": "green",
  "indices": {
    "count": 212,
    "shards": {
      "total": 1191,
      "primaries": 595,
      "replication": 1.0016806722689076,
      "index": {
        "shards": {
          "min": 2,
          "max": 30,
          "avg": 5.617924528301887
        },
        "primaries": {
          "min": 1,
          "max": 15,
          "avg": 2.806603773584906
        },
        "replication": {
          "min": 1,
          "max": 2,
          "avg": 1.0047169811320755
        }
      }
    },
    "docs": {
      "count": 22693559250,
      "deleted": 494
    },
    "store": {
      "size_in_bytes": 37214680604546,
      "total_data_set_size_in_bytes": 37214680604546,
      "reserved_in_bytes": 0
    },
    "fielddata": {
      "memory_size_in_bytes": 2568648,
      "evictions": 0,
      "global_ordinals": {
        "build_time_in_millis": 41434
      }
    },
    "query_cache": {
      "memory_size_in_bytes": 1923327463,
      "total_count": 22929521,
      "hit_count": 158258,
      "miss_count": 22771263,
      "cache_size": 12592,
      "cache_count": 13576,
      "evictions": 984
    },
    "completion": {
      "size_in_bytes": 0
    },
    "segments": {
      "count": 35826,
      "memory_in_bytes": 0,
      "terms_memory_in_bytes": 0,
      "stored_fields_memory_in_bytes": 0,
      "term_vectors_memory_in_bytes": 0,
      "norms_memory_in_bytes": 0,
      "points_memory_in_bytes": 0,
      "doc_values_memory_in_bytes": 0,
      "index_writer_memory_in_bytes": 1065343264,
      "version_map_memory_in_bytes": 23167853,
      "fixed_bit_set_memory_in_bytes": 1302744,
      "max_unsafe_auto_id_timestamp": 1702272981490,
      "file_sizes": {}
    },
    "mappings": {
      "total_field_count": 574492,
      "total_deduplicated_field_count": 523593,
      "total_deduplicated_mapping_size_in_bytes": 2815893,
      "field_types": [
        {
          "name": "alias",
          "count": 3482,
          "index_count": 57,
          "script_count": 0
        },
        {
          "name": "binary",
          "count": 2,
          "index_count": 2,
          "script_count": 0
        },
        {
          "name": "boolean",
          "count": 8072,
          "index_count": 182,
          "script_count": 0
        },
        {
          "name": "byte",
          "count": 10,
          "index_count": 10,
          "script_count": 0
        },
        {
          "name": "constant_keyword",
          "count": 478,
          "index_count": 154,
          "script_count": 0
        },
        {
          "name": "date",
          "count": 10845,
          "index_count": 187,
          "script_count": 0
        },
        {
          "name": "date_nanos",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "date_range",
          "count": 8,
          "index_count": 8,
          "script_count": 0
        },
        {
          "name": "double",
          "count": 4110,
          "index_count": 140,
          "script_count": 0
        },
        {
          "name": "double_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "flattened",
          "count": 2544,
          "index_count": 136,
          "script_count": 0
        },
        {
          "name": "float",
          "count": 7005,
          "index_count": 147,
          "script_count": 0
        },
        {
          "name": "float_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "geo_point",
          "count": 633,
          "index_count": 133,
          "script_count": 0
        },
        {
          "name": "geo_shape",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "half_float",
          "count": 51,
          "index_count": 11,
          "script_count": 0
        },
        {
          "name": "integer",
          "count": 11,
          "index_count": 7,
          "script_count": 0
        },
        {
          "name": "integer_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "ip",
          "count": 8125,
          "index_count": 158,
          "script_count": 0
        },
        {
          "name": "ip_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "keyword",
          "count": 276472,
          "index_count": 188,
          "script_count": 0
        },
        {
          "name": "long",
          "count": 118501,
          "index_count": 171,
          "script_count": 0
        },
        {
          "name": "long_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "match_only_text",
          "count": 5442,
          "index_count": 155,
          "script_count": 0
        },
        {
          "name": "nested",
          "count": 1226,
          "index_count": 141,
          "script_count": 0
        },
        {
          "name": "object",
          "count": 95639,
          "index_count": 185,
          "script_count": 0
        },
        {
          "name": "rank_features",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "scaled_float",
          "count": 1173,
          "index_count": 137,
          "script_count": 0
        },
        {
          "name": "shape",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "short",
          "count": 8933,
          "index_count": 45,
          "script_count": 0
        },
        {
          "name": "text",
          "count": 20027,
          "index_count": 150,
          "script_count": 0
        },
        {
          "name": "version",
          "count": 14,
          "index_count": 14,
          "script_count": 0
        },
        {
          "name": "wildcard",
          "count": 1680,
          "index_count": 146,
          "script_count": 0
        }
      ],
      "runtime_field_types": []
    },
    "analysis": {
      "char_filter_types": [],
      "tokenizer_types": [],
      "filter_types": [],
      "analyzer_types": [],
      "built_in_char_filters": [],
      "built_in_tokenizers": [],
      "built_in_filters": [],
      "built_in_analyzers": [],
      "synonyms": {}
    },
    "versions": [
      {
        "version": "8080299",
        "index_count": 33,
        "primary_shard_count": 33,
        "total_primary_bytes": 536580610
      },
      {
        "version": "8090299",
        "index_count": 2,
        "primary_shard_count": 2,
        "total_primary_bytes": 236347555
      },
      {
        "version": "8100499",
        "index_count": 177,
        "primary_shard_count": 560,
        "total_primary_bytes": 18610864409529
      }
    ],
    "search": {
      "total": 4656254,
      "queries": {
        "bool": 4489062,
        "prefix": 9,
        "match": 217466,
        "range": 3924950,
        "nested": 12085,
        "multi_match": 82,
        "wildcard": 1140,
        "match_phrase": 3098527,
        "terms": 558612,
        "match_all": 14,
        "ids": 29,
        "match_phrase_prefix": 94,
        "exists": 455826,
        "term": 1137246,
        "query_string": 681,
        "simple_query_string": 52047
      },
      "sections": {
        "search_after": 3,
        "highlight": 869,
        "stored_fields": 1707,
        "runtime_mappings": 321238,
        "query": 4572121,
        "script_fields": 1833,
        "terminate_after": 252,
        "_source": 103564,
        "pit": 25495,
        "fields": 28872,
        "collapse": 135985,
        "aggs": 541466
      }
    },
    "dense_vector": {
      "value_count": 0
    }
  },
  "nodes": {
    "count": {
      "total": 19,
      "coordinating_only": 0,
      "data": 0,
      "data_cold": 5,
      "data_content": 6,
      "data_frozen": 0,
      "data_hot": 6,
      "data_warm": 5,
      "index": 0,
      "ingest": 6,
      "master": 3,
      "ml": 0,
      "remote_cluster_client": 0,
      "search": 0,
      "transform": 0,
      "voting_only": 0
    },
    "versions": [
      "8.10.4"
    ],
    "os": {
      "available_processors": 80,
      "allocated_processors": 80,
      "names": [
        {
          "name": "Linux",
          "count": 19
        }
      ],
      "pretty_names": [
        {
          "pretty_name": "Ubuntu 20.04.6 LTS",
          "count": 19
        }
      ],
      "architectures": [
        {
          "arch": "amd64",
          "count": 19
        }
      ],
      "mem": {
        "total_in_bytes": 865435910144,
        "adjusted_total_in_bytes": 865435910144,
        "free_in_bytes": 31167352832,
        "used_in_bytes": 834268557312,
        "free_percent": 4,
        "used_percent": 96
      }
    },
    "process": {
      "cpu": {
        "percent": 334
      },
      "open_file_descriptors": {
        "min": 982,
        "max": 16590,
        "avg": 5750
      }
    },
    "jvm": {
      "max_uptime_in_millis": 1828392825,
      "versions": [
        {
          "version": "21",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "vm_version": "21+35-2513",
          "vm_vendor": "Oracle Corporation",
          "bundled_jdk": true,
          "using_bundled_jdk": true,
          "count": 19
        }
      ],
      "mem": {
        "heap_used_in_bytes": 126145158968,
        "heap_max_in_bytes": 434655723520
      },
      "threads": 1749
    },
    "fs": {
      "total_in_bytes": 64165201412096,
      "free_in_bytes": 26819467804672,
      "available_in_bytes": 26819149037568
    },
    "plugins": [
      {
        "name": "mapper-size",
        "version": "8.10.4",
        "elasticsearch_version": "8.10.4",
        "java_version": "17",
        "description": "The Mapper Size plugin allows document to record their uncompressed size at index time.",
        "classname": "org.elasticsearch.plugin.mapper.MapperSizePlugin",
        "extended_plugins": [],
        "has_native_controller": false,
        "licensed": false,
        "is_official": true
      }
    ],
    "network_types": {
      "transport_types": {
        "security4": 19
      },
      "http_types": {
        "security4": 19
      }
    },
    "discovery_types": {
      "multi-node": 19
    },
    "packaging_types": [
      {
        "flavor": "default",
        "type": "docker",
        "count": 19
      }
    ],
    "ingest": {
      "number_of_pipelines": 177,
      "processor_stats": {
        "append": {
          "count": 23,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "attachment": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "convert": {
          "count": 149287,
          "failed": 0,
          "current": 0,
          "time_in_millis": 298
        },
        "date": {
          "count": 2188293,
          "failed": 0,
          "current": 0,
          "time_in_millis": 48994
        },
        "date_index_name": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "dot_expander": {
          "count": 654263,
          "failed": 0,
          "current": 0,
          "time_in_millis": 6612
        },
        "drop": {
          "count": 344712,
          "failed": 0,
          "current": 0,
          "time_in_millis": 62
        },
        "fail": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "fingerprint": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "foreach": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "geoip": {
          "count": 8572682628,
          "failed": 0,
          "current": 0,
          "time_in_millis": 11864666
        },
        "grok": {
          "count": 833166,
          "failed": 158569,
          "current": 0,
          "time_in_millis": 12148
        },
        "gsub": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "inference": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "join": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "json": {
          "count": 658530,
          "failed": 91,
          "current": 0,
          "time_in_millis": 30864
        },
        "pipeline": {
          "count": 3201775,
          "failed": 0,
          "current": 0,
          "time_in_millis": 95657
        },
        "remove": {
          "count": 5081413,
          "failed": 764,
          "current": 0,
          "time_in_millis": 7808
        },
        "rename": {
          "count": 12859302333,
          "failed": 0,
          "current": 0,
          "time_in_millis": 5546931
        },
        "script": {
          "count": 4578564372,
          "failed": 86,
          "current": 0,
          "time_in_millis": 32388611
        },
        "set": {
          "count": 12861104422,
          "failed": 0,
          "current": 0,
          "time_in_millis": 8515263
        },
        "set_security_user": {
          "count": 2173180,
          "failed": 0,
          "current": 0,
          "time_in_millis": 24180
        },
        "split": {
          "count": 24,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "trim": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "uri_parts": {
          "count": 6,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "user_agent": {
          "count": 4286341314,
          "failed": 0,
          "current": 0,
          "time_in_millis": 3871032
        }
      }
    },
    "indexing_pressure": {
      "memory": {
        "current": {
          "combined_coordinating_and_primary_in_bytes": 0,
          "coordinating_in_bytes": 0,
          "primary_in_bytes": 0,
          "replica_in_bytes": 0,
          "all_in_bytes": 0
        },
        "total": {
          "combined_coordinating_and_primary_in_bytes": 0,
          "coordinating_in_bytes": 0,
          "primary_in_bytes": 0,
          "replica_in_bytes": 0,
          "all_in_bytes": 0,
          "coordinating_rejections": 0,
          "primary_rejections": 0,
          "replica_rejections": 0
        },
        "limit_in_bytes": 0
      }
    }
  },
  "snapshots": {
    "current_counts": {
      "snapshots": 0,
      "shard_snapshots": 0,
      "snapshot_deletions": 0,
      "concurrent_operations": 0,
      "cleanups": 0
    },
    "repositories": {}
  }
}

Why are you rolling over every 2 hours?

just to have more predictable amount in indexes (shards) in each level (hot,warm,cold). Is it bad practice?

How many primary and replica shards does each index have during indexing?

4 primaries + 1 replica, as it's rollover max_age 2h it generates ~8 indexes per day = 2.8 TB including replica

4

It can result in a lot of small shards, so is not something I have seen frequently.

If the max age is 2 hours it should generate 12 indices per day. If that correspond to2.8TB in 96 shards (1242) you have good size shards.

It sounds like your hot nodes are indexing a lot every day and that means that there is a lot of data to transfer over when the warm phase transition hits. I am not sure what impact this would have on heap usage, but it should correlate with the memory spikes you are seeing. I am not aware of any way to reduce this impact.

5

maybe you can advice when better not to use ILM (hot,warm,cold) architecture? Because looks that all data transfer between tiers in such architecture require more resources and utilisation is not predictable because of spikes comparing to approach when only data nodes are used without tiers

for example I saw comment that ILM was disabled for APM by default apmpackage: support phases (warm, cold) in ILM in apm package · Issue #11579 · elastic/apm-server · GitHub

how can we understand that ILM (multi tier levels) is more effective? :slight_smile:

6

You are transferring a lot of data each day from the hot nodes. It need to go over the network onto the warm nodes (fewer than hot nodes), which I assume have slower storage. The same amount of data is then moved from the warm nodes every day to the cold nodes, which will further add load to the warm and cold nodes. I wonder if it may not be better to just have a hot and warm tier. That way each warm node would hold more data but move a lot less data in and out every day.

1 Like
7

thank you!

also what do you think about start using actions during "hot" -> "warm"

"actions": {
            "forcemerge": {
              "max_num_segments": 1
            },
            "shrink": {
              "number_of_shards": 1
            }
          }

does it increase cpu/memory usage even more? Or anyway all data already processed and resource consumption should be almost the same but as result we receive more efficient and compact storage in "warm" nodes?

8

Given that you shards, if i calculate correctly, are reasonably large i would not shrink. Forcemerging can reduce heap usage but is very I/O intensive, so i would probably skip that too.

I do not believe that is the case.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.