# Memory spikes on ILM transition
**URL:** https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008
**Category:** Elasticsearch
**Tags:** ilm-index-lifecycle-management
**Created:** [December 11, 2023, 5:21am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008 "2023-12-11T05:21:30Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@azhurbilo](https://discuss.elastic.co/u/azhurbilo)
#### Post date: [December 11, 2023, 5:21am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008/1 "2023-12-11T05:21:30Z")
We use hot-warm-cold architecture
policy
```auto
{
"filebeat": {
"version": 39128,
"modified_date": "2023-12-11T04:05:20.404Z",
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "2h",
"max_primary_shard_size": "50gb"
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "1h",
"actions": {
"allocate": {
"number_of_replicas": 1,
"include": {},
"exclude": {},
"require": {}
},
"readonly": {},
"set_priority": {
"priority": 50
}
}
},
"cold": {
"min_age": "1d",
"actions": {
"allocate": {
"number_of_replicas": 1,
"include": {},
"exclude": {},
"require": {}
},
"readonly": {},
"set_priority": {
"priority": 0
}
}
},
"delete": {
"min_age": "11d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}
}
```
in "hot" initially we also write "number\_of\_replicas": 1 for index
and as we don't use any transform actions like "forcemerge" or "shrink" we expect that it's just network copy from "hot" nodes to "warm" nodes, but according metrics we see spikes in network and spikes in memory (off heap) which is problem for us as it always triggers memory alerts \>90% usage.
Can someone advice what do to to prevent such memory spikes?
our hot node setup
```auto
resources:
limits:
cpu: '8'
memory: 48Gi
requests:
cpu: '6'
memory: 48Gi
```
Xms/Xmx/MaxDirectMemorySize default auto values generated by Elasticsearch itself
And maybe some general ILM good practices, maybe we need change something in our policy?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 11, 2023, 5:34am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008/2 "2023-12-11T05:34:20Z")
What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/cluster-stats.html)?
> [@azhurbilo](#):
>
> ```auto
> "hot": {
> "min_age": "0ms",
> "actions": {
> "rollover": {
> "max_age": "2h",
> "max_primary_shard_size": "50gb"
>
> ```
Why are you rolling over every 2 hours? How many primary and replica shards does each index have during indexing? How much data are you ingesting per day?
---
### Author:  [@azhurbilo](https://discuss.elastic.co/u/azhurbilo)
#### Post date: [December 11, 2023, 7:44am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008/3 "2023-12-11T07:44:22Z")
cluster stats
```auto
{
"_nodes": {
"total": 19,
"successful": 19,
"failed": 0
},
"cluster_name": "elastic-search",
"cluster_uuid": "xxxxxx",
"timestamp": 1702280147268,
"status": "green",
"indices": {
"count": 212,
"shards": {
"total": 1191,
"primaries": 595,
"replication": 1.0016806722689076,
"index": {
"shards": {
"min": 2,
"max": 30,
"avg": 5.617924528301887
},
"primaries": {
"min": 1,
"max": 15,
"avg": 2.806603773584906
},
"replication": {
"min": 1,
"max": 2,
"avg": 1.0047169811320755
}
}
},
"docs": {
"count": 22693559250,
"deleted": 494
},
"store": {
"size_in_bytes": 37214680604546,
"total_data_set_size_in_bytes": 37214680604546,
"reserved_in_bytes": 0
},
"fielddata": {
"memory_size_in_bytes": 2568648,
"evictions": 0,
"global_ordinals": {
"build_time_in_millis": 41434
}
},
"query_cache": {
"memory_size_in_bytes": 1923327463,
"total_count": 22929521,
"hit_count": 158258,
"miss_count": 22771263,
"cache_size": 12592,
"cache_count": 13576,
"evictions": 984
},
"completion": {
"size_in_bytes": 0
},
"segments": {
"count": 35826,
"memory_in_bytes": 0,
"terms_memory_in_bytes": 0,
"stored_fields_memory_in_bytes": 0,
"term_vectors_memory_in_bytes": 0,
"norms_memory_in_bytes": 0,
"points_memory_in_bytes": 0,
"doc_values_memory_in_bytes": 0,
"index_writer_memory_in_bytes": 1065343264,
"version_map_memory_in_bytes": 23167853,
"fixed_bit_set_memory_in_bytes": 1302744,
"max_unsafe_auto_id_timestamp": 1702272981490,
"file_sizes": {}
},
"mappings": {
"total_field_count": 574492,
"total_deduplicated_field_count": 523593,
"total_deduplicated_mapping_size_in_bytes": 2815893,
"field_types": [
{
"name": "alias",
"count": 3482,
"index_count": 57,
"script_count": 0
},
{
"name": "binary",
"count": 2,
"index_count": 2,
"script_count": 0
},
{
"name": "boolean",
"count": 8072,
"index_count": 182,
"script_count": 0
},
{
"name": "byte",
"count": 10,
"index_count": 10,
"script_count": 0
},
{
"name": "constant_keyword",
"count": 478,
"index_count": 154,
"script_count": 0
},
{
"name": "date",
"count": 10845,
"index_count": 187,
"script_count": 0
},
{
"name": "date_nanos",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "date_range",
"count": 8,
"index_count": 8,
"script_count": 0
},
{
"name": "double",
"count": 4110,
"index_count": 140,
"script_count": 0
},
{
"name": "double_range",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "flattened",
"count": 2544,
"index_count": 136,
"script_count": 0
},
{
"name": "float",
"count": 7005,
"index_count": 147,
"script_count": 0
},
{
"name": "float_range",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "geo_point",
"count": 633,
"index_count": 133,
"script_count": 0
},
{
"name": "geo_shape",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "half_float",
"count": 51,
"index_count": 11,
"script_count": 0
},
{
"name": "integer",
"count": 11,
"index_count": 7,
"script_count": 0
},
{
"name": "integer_range",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "ip",
"count": 8125,
"index_count": 158,
"script_count": 0
},
{
"name": "ip_range",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "keyword",
"count": 276472,
"index_count": 188,
"script_count": 0
},
{
"name": "long",
"count": 118501,
"index_count": 171,
"script_count": 0
},
{
"name": "long_range",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "match_only_text",
"count": 5442,
"index_count": 155,
"script_count": 0
},
{
"name": "nested",
"count": 1226,
"index_count": 141,
"script_count": 0
},
{
"name": "object",
"count": 95639,
"index_count": 185,
"script_count": 0
},
{
"name": "rank_features",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "scaled_float",
"count": 1173,
"index_count": 137,
"script_count": 0
},
{
"name": "shape",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "short",
"count": 8933,
"index_count": 45,
"script_count": 0
},
{
"name": "text",
"count": 20027,
"index_count": 150,
"script_count": 0
},
{
"name": "version",
"count": 14,
"index_count": 14,
"script_count": 0
},
{
"name": "wildcard",
"count": 1680,
"index_count": 146,
"script_count": 0
}
],
"runtime_field_types": []
},
"analysis": {
"char_filter_types": [],
"tokenizer_types": [],
"filter_types": [],
"analyzer_types": [],
"built_in_char_filters": [],
"built_in_tokenizers": [],
"built_in_filters": [],
"built_in_analyzers": [],
"synonyms": {}
},
"versions": [
{
"version": "8080299",
"index_count": 33,
"primary_shard_count": 33,
"total_primary_bytes": 536580610
},
{
"version": "8090299",
"index_count": 2,
"primary_shard_count": 2,
"total_primary_bytes": 236347555
},
{
"version": "8100499",
"index_count": 177,
"primary_shard_count": 560,
"total_primary_bytes": 18610864409529
}
],
"search": {
"total": 4656254,
"queries": {
"bool": 4489062,
"prefix": 9,
"match": 217466,
"range": 3924950,
"nested": 12085,
"multi_match": 82,
"wildcard": 1140,
"match_phrase": 3098527,
"terms": 558612,
"match_all": 14,
"ids": 29,
"match_phrase_prefix": 94,
"exists": 455826,
"term": 1137246,
"query_string": 681,
"simple_query_string": 52047
},
"sections": {
"search_after": 3,
"highlight": 869,
"stored_fields": 1707,
"runtime_mappings": 321238,
"query": 4572121,
"script_fields": 1833,
"terminate_after": 252,
"_source": 103564,
"pit": 25495,
"fields": 28872,
"collapse": 135985,
"aggs": 541466
}
},
"dense_vector": {
"value_count": 0
}
},
"nodes": {
"count": {
"total": 19,
"coordinating_only": 0,
"data": 0,
"data_cold": 5,
"data_content": 6,
"data_frozen": 0,
"data_hot": 6,
"data_warm": 5,
"index": 0,
"ingest": 6,
"master": 3,
"ml": 0,
"remote_cluster_client": 0,
"search": 0,
"transform": 0,
"voting_only": 0
},
"versions": [
"8.10.4"
],
"os": {
"available_processors": 80,
"allocated_processors": 80,
"names": [
{
"name": "Linux",
"count": 19
}
],
"pretty_names": [
{
"pretty_name": "Ubuntu 20.04.6 LTS",
"count": 19
}
],
"architectures": [
{
"arch": "amd64",
"count": 19
}
],
"mem": {
"total_in_bytes": 865435910144,
"adjusted_total_in_bytes": 865435910144,
"free_in_bytes": 31167352832,
"used_in_bytes": 834268557312,
"free_percent": 4,
"used_percent": 96
}
},
"process": {
"cpu": {
"percent": 334
},
"open_file_descriptors": {
"min": 982,
"max": 16590,
"avg": 5750
}
},
"jvm": {
"max_uptime_in_millis": 1828392825,
"versions": [
{
"version": "21",
"vm_name": "OpenJDK 64-Bit Server VM",
"vm_version": "21+35-2513",
"vm_vendor": "Oracle Corporation",
"bundled_jdk": true,
"using_bundled_jdk": true,
"count": 19
}
],
"mem": {
"heap_used_in_bytes": 126145158968,
"heap_max_in_bytes": 434655723520
},
"threads": 1749
},
"fs": {
"total_in_bytes": 64165201412096,
"free_in_bytes": 26819467804672,
"available_in_bytes": 26819149037568
},
"plugins": [
{
"name": "mapper-size",
"version": "8.10.4",
"elasticsearch_version": "8.10.4",
"java_version": "17",
"description": "The Mapper Size plugin allows document to record their uncompressed size at index time.",
"classname": "org.elasticsearch.plugin.mapper.MapperSizePlugin",
"extended_plugins": [],
"has_native_controller": false,
"licensed": false,
"is_official": true
}
],
"network_types": {
"transport_types": {
"security4": 19
},
"http_types": {
"security4": 19
}
},
"discovery_types": {
"multi-node": 19
},
"packaging_types": [
{
"flavor": "default",
"type": "docker",
"count": 19
}
],
"ingest": {
"number_of_pipelines": 177,
"processor_stats": {
"append": {
"count": 23,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"attachment": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"convert": {
"count": 149287,
"failed": 0,
"current": 0,
"time_in_millis": 298
},
"date": {
"count": 2188293,
"failed": 0,
"current": 0,
"time_in_millis": 48994
},
"date_index_name": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"dot_expander": {
"count": 654263,
"failed": 0,
"current": 0,
"time_in_millis": 6612
},
"drop": {
"count": 344712,
"failed": 0,
"current": 0,
"time_in_millis": 62
},
"fail": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"fingerprint": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"foreach": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"geoip": {
"count": 8572682628,
"failed": 0,
"current": 0,
"time_in_millis": 11864666
},
"grok": {
"count": 833166,
"failed": 158569,
"current": 0,
"time_in_millis": 12148
},
"gsub": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"inference": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"join": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"json": {
"count": 658530,
"failed": 91,
"current": 0,
"time_in_millis": 30864
},
"pipeline": {
"count": 3201775,
"failed": 0,
"current": 0,
"time_in_millis": 95657
},
"remove": {
"count": 5081413,
"failed": 764,
"current": 0,
"time_in_millis": 7808
},
"rename": {
"count": 12859302333,
"failed": 0,
"current": 0,
"time_in_millis": 5546931
},
"script": {
"count": 4578564372,
"failed": 86,
"current": 0,
"time_in_millis": 32388611
},
"set": {
"count": 12861104422,
"failed": 0,
"current": 0,
"time_in_millis": 8515263
},
"set_security_user": {
"count": 2173180,
"failed": 0,
"current": 0,
"time_in_millis": 24180
},
"split": {
"count": 24,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"trim": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"uri_parts": {
"count": 6,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"user_agent": {
"count": 4286341314,
"failed": 0,
"current": 0,
"time_in_millis": 3871032
}
}
},
"indexing_pressure": {
"memory": {
"current": {
"combined_coordinating_and_primary_in_bytes": 0,
"coordinating_in_bytes": 0,
"primary_in_bytes": 0,
"replica_in_bytes": 0,
"all_in_bytes": 0
},
"total": {
"combined_coordinating_and_primary_in_bytes": 0,
"coordinating_in_bytes": 0,
"primary_in_bytes": 0,
"replica_in_bytes": 0,
"all_in_bytes": 0,
"coordinating_rejections": 0,
"primary_rejections": 0,
"replica_rejections": 0
},
"limit_in_bytes": 0
}
}
},
"snapshots": {
"current_counts": {
"snapshots": 0,
"shard_snapshots": 0,
"snapshot_deletions": 0,
"concurrent_operations": 0,
"cleanups": 0
},
"repositories": {}
}
}
```
> Why are you rolling over every 2 hours?
just to have more predictable amount in indexes (shards) in each level (hot,warm,cold). Is it bad practice?
> How many primary and replica shards does each index have during indexing?
4 primaries + 1 replica, as it's rollover max\_age 2h it generates ~8 indexes per day = 2.8 TB including replica
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 11, 2023, 8:01am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008/4 "2023-12-11T08:01:48Z")
> [@azhurbilo](#):
>
> just to have more predictable amount in indexes (shards) in each level (hot,warm,cold). Is it bad practice?
It can result in a lot of small shards, so is not something I have seen frequently.
> [@azhurbilo](#):
>
> 4 primaries + 1 replica, as it's rollover max\_age 2h it generates ~8 indexes per day = 2.8 TB including replica
If the max age is 2 hours it should generate 12 indices per day. If that correspond to2.8TB in 96 shards (12_4_2) you have good size shards.
It sounds like your hot nodes are indexing a lot every day and that means that there is a lot of data to transfer over when the warm phase transition hits. I am not sure what impact this would have on heap usage, but it should correlate with the memory spikes you are seeing. I am not aware of any way to reduce this impact.
---
### Author:  [@azhurbilo](https://discuss.elastic.co/u/azhurbilo)
#### Post date: [December 11, 2023, 9:10am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008/5 "2023-12-11T09:10:58Z")
> [@Christian\_Dahlqvist](#):
>
> I am not aware of any way to reduce this impact.
maybe you can advice when better not to use ILM (hot,warm,cold) architecture? Because looks that all data transfer between tiers in such architecture require more resources and utilisation is not predictable because of spikes comparing to approach when only data nodes are used without tiers
for example I saw comment that ILM was disabled for APM by default [apmpackage: support phases (warm, cold) in ILM in apm package · Issue #11579 · elastic/apm-server · GitHub](https://github.com/elastic/apm-server/issues/11579#issuecomment-1705786787)
how can we understand that ILM (multi tier levels) is more effective? 🙂
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 11, 2023, 10:44am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008/6 "2023-12-11T10:44:43Z")
You are transferring a lot of data each day from the hot nodes. It need to go over the network onto the warm nodes (fewer than hot nodes), which I assume have slower storage. The same amount of data is then moved from the warm nodes every day to the cold nodes, which will further add load to the warm and cold nodes. I wonder if it may not be better to just have a hot and warm tier. That way each warm node would hold more data but move a lot less data in and out every day.
---
### Author:  [@azhurbilo](https://discuss.elastic.co/u/azhurbilo)
#### Post date: [December 11, 2023, 11:17am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008/7 "2023-12-11T11:17:08Z")
thank you!
also what do you think about start using actions during "hot" -\> "warm"
```auto
"actions": {
"forcemerge": {
"max_num_segments": 1
},
"shrink": {
"number_of_shards": 1
}
}
```
does it increase cpu/memory usage even more? Or anyway all data already processed and resource consumption should be almost the same but as result we receive more efficient and compact storage in "warm" nodes?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 11, 2023, 11:38am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008/8 "2023-12-11T11:38:24Z")
Given that you shards, if i calculate correctly, are reasonably large i would not shrink. Forcemerging can reduce heap usage but is very I/O intensive, so i would probably skip that too.
> [@azhurbilo](#):
>
> we expect that it's just network copy from "hot" nodes to "warm" nodes
I do not believe that is the case.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 8, 2024, 11:38am UTC](https://discuss.elastic.co/t/memory-spikes-on-ilm-transition/349008/9 "2024-01-08T11:38:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.