# Out Of Memory crash, few documents & load

**URL:** <https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597>\
**Category:** Elasticsearch\
**Created:** [October 13, 2021, 8:55am UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597 "2021-10-13T08:55:03Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![cyril.r](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@cyril.r](https://discuss.elastic.co/u/cyril.r)\
**Post date:** [October 13, 2021, 8:55am UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/1 "2021-10-13T08:55:03Z")

</div>

Hi there,

I have a small Elasticsearch single node cluster for development purpose that is crashing about one time per month, even when there are no trafic/requests (like in the middle of the night, no one is working on it).

It is running Elasticsearch 7.4.2 on a dualcore instance with 2GB RAM.  
According to kibana (also running on the same node), there are

- 4,496,347 documents
- 77 indices
- 175 primary shards
- 2.2Gb of disk usage

The document count is mostly the .monitoring indices, we have \< 10000 documents on from our own.

The node performs well without any issues, until it suddenly crashes.

JVM config is :

```auto
-Xms768m
-Xmx1g

```

Here are the output from `_cluster/stats?human&pretty` (after the restart):

```auto
{
    "_nodes": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "cluster_name": "elasticsearch",
    "cluster_uuid": "WJOOuxXuTd2dQ0UMMhuPkg",
    "timestamp": 1634113459473,
    "status": "yellow",
    "indices": {
        "count": 77,
        "shards": {
            "total": 175,
            "primaries": 175,
            "replication": 0.0,
            "index": {
                "shards": {
                    "min": 1,
                    "max": 5,
                    "avg": 2.272727272727273
                },
                "primaries": {
                    "min": 1,
                    "max": 5,
                    "avg": 2.272727272727273
                },
                "replication": {
                    "min": 0.0,
                    "max": 0.0,
                    "avg": 0.0
                }
            }
        },
        "docs": {
            "count": 4498732,
            "deleted": 2945166
        },
        "store": {
            "size": "2.1gb",
            "size_in_bytes": 2342333611
        },
        "fielddata": {
            "memory_size": "21.6kb",
            "memory_size_in_bytes": 22128,
            "evictions": 0
        },
        "query_cache": {
            "memory_size": "1.8mb",
            "memory_size_in_bytes": 1921056,
            "total_count": 31965,
            "hit_count": 13810,
            "miss_count": 18155,
            "cache_size": 276,
            "cache_count": 330,
            "evictions": 54
        },
        "completion": {
            "size": "0b",
            "size_in_bytes": 0
        },
        "segments": {
            "count": 486,
            "memory": "5.1mb",
            "memory_in_bytes": 5358682,
            "terms_memory": "2.7mb",
            "terms_memory_in_bytes": 2900928,
            "stored_fields_memory": "507.9kb",
            "stored_fields_memory_in_bytes": 520096,
            "term_vectors_memory": "0b",
            "term_vectors_memory_in_bytes": 0,
            "norms_memory": "48.5kb",
            "norms_memory_in_bytes": 49664,
            "points_memory": "1mb",
            "points_memory_in_bytes": 1123210,
            "doc_values_memory": "746.8kb",
            "doc_values_memory_in_bytes": 764784,
            "index_writer_memory": "0b",
            "index_writer_memory_in_bytes": 0,
            "version_map_memory": "0b",
            "version_map_memory_in_bytes": 0,
            "fixed_bit_set": "905.6kb",
            "fixed_bit_set_memory_in_bytes": 927416,
            "max_unsafe_auto_id_timestamp": 1634108911687,
            "file_sizes": {}
        }
    },
    "nodes": {
        "count": {
            "total": 1,
            "coordinating_only": 0,
            "data": 1,
            "ingest": 1,
            "master": 1,
            "ml": 1,
            "voting_only": 0
        },
        "versions": [
            "7.4.2"
        ],
        "os": {
            "available_processors": 2,
            "allocated_processors": 2,
            "names": [
                {
                    "name": "Linux",
                    "count": 1
                }
            ],
            "pretty_names": [
                {
                    "pretty_name": "Ubuntu 20.04.1 LTS",
                    "count": 1
                }
            ],
            "mem": {
                "total": "1.9gb",
                "total_in_bytes": 2044534784,
                "free": "85.6mb",
                "free_in_bytes": 89849856,
                "used": "1.8gb",
                "used_in_bytes": 1954684928,
                "free_percent": 4,
                "used_percent": 96
            }
        },
        "process": {
            "cpu": {
                "percent": 8
            },
            "open_file_descriptors": {
                "min": 1353,
                "max": 1353,
                "avg": 1353
            }
        },
        "jvm": {
            "max_uptime": "1.2h",
            "max_uptime_in_millis": 4592145,
            "versions": [
                {
                    "version": "13.0.1",
                    "vm_name": "OpenJDK 64-Bit Server VM",
                    "vm_version": "13.0.1+9",
                    "vm_vendor": "AdoptOpenJDK",
                    "bundled_jdk": true,
                    "using_bundled_jdk": true,
                    "count": 1
                }
            ],
            "mem": {
                "heap_used": "535mb",
                "heap_used_in_bytes": 561009184,
                "heap_max": "1007.3mb",
                "heap_max_in_bytes": 1056309248
            },
            "threads": 48
        },
        "fs": {
            "total": "67.7gb",
            "total_in_bytes": 72794869760,
            "free": "58.7gb",
            "free_in_bytes": 63081897984,
            "available": "58.7gb",
            "available_in_bytes": 63065120768
        },
        "plugins": [
            {
                "name": "repository-s3",
                "version": "7.4.2",
                "elasticsearch_version": "7.4.2",
                "java_version": "1.8",
                "description": "The S3 repository plugin adds S3 repositories",
                "classname": "org.elasticsearch.repositories.s3.S3RepositoryPlugin",
                "extended_plugins": [],
                "has_native_controller": false
            }
        ],
        "network_types": {
            "transport_types": {
                "security4": 1
            },
            "http_types": {
                "security4": 1
            }
        },
        "discovery_types": {
            "single-node": 1
        },
        "packaging_types": [
            {
                "flavor": "default",
                "type": "deb",
                "count": 1
            }
        ]
    }
}

```

(I knowI have replica shards that are unassigned, but it shouldn't be an issue)

There are nothing on the logs when the crash happens :

```auto
#Note: log time is UTC
[2021-10-13T01:30:00,010][INFO][o.e.x.m.a.TransportDeleteExpiredDataAction] [flus-es-dev] Deleting expired data
[2021-10-13T01:30:00,024][INFO][o.e.x.m.a.TransportDeleteExpiredDataAction] [flus-es-dev] Completed deletion of expired ML data
[2021-10-13T01:30:00,024][INFO][o.e.x.m.MlDailyMaintenanceService] [flus-es-dev] Successfully completed [ML] maintenance tasks

--- it crashed at Oct 13 03:17:23 according to the kernel log OOM ---
-> restart
[2021-10-13T07:07:51,910][INFO][o.e.e.NodeEnvironment] [flus-es-dev] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [58.5gb], net total_space [67.7gb], types [ext4]
[2021-10-13T07:07:51,937][INFO][o.e.e.NodeEnvironment] [flus-es-dev] heap size [1007.3mb], compressed ordinary object pointers [true]
[...]

```

The only thing that seems strange is the gc.log, it is constantly running with "allocation failed" messages, I read elsewhere that it shouldn't be an issue but I find it strange.

--\> **See attached GC log : [[2021-10-13T08:34:59.517+0000][468][gc,start] GC(596) Pause Young (Allocati - Pastebin.com](https://pastebin.com/YUfZSwwb)**

And kibana capture (times are UTC+2 Paris) :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/8/f8550af8d39d3ab884b4f1bdbab2e3e4f90b0a23.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/3/036abc50372f83a69ac34096fc915c33c103105a.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af8d092028b1db118ffee33ec20680308ee9bcef.png)

Any clues?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 13, 2021, 10:35am UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/2 "2021-10-13T10:35:06Z")

</div>

1gb of HEAP is generally good for up to 20 shards.  
Here, you would need something like 9gb of HEAP for around 180 shards.

```auto
-Xms768m
-Xmx1g

```

You should use the same value here. I'm surprised that you can even start Elasticsearch. But may be it's only a WARN as you are not in production. You should see this check happening: [Heap size check | Elasticsearch Guide [7.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/_heap_size_check.html)

Advices:

- Increase the HEAP size and if possible the available RAM
- Or remove non needed indices. (the monitoring ones...)
- Upgrade to 7.15.0. A lot of things happened in the mean time.

---

<div class="post-metadata">

**Author:** ![cyril.r](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@cyril.r](https://discuss.elastic.co/u/cyril.r)\
**Post date:** [October 13, 2021, 12:20pm UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/3 "2021-10-13T12:20:30Z")

</div>

Hello @dadoonet ,

Thanks for the clues, I will change the Xms value but indeed that's not the source of the issue.

If I understand correctly, I should allocate at least 9GB (so about 1GB for 20 shards) of JVM heap so I need to have 18GB of memory since 50% is for the JVM and 50% for the system ?  
That's a lot of memory.

What could be causing the OOM error while there are no trafic and no snapshots running?

Does the upgrade to 7.15.0, changes something regarding the memory that would eliminate this issue?

I have to provide a strong and concise answer to my manager, I understand according to [Size your shards | Elasticsearch Guide [7.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html) that we are oversharding, but the point of my manager is "it is working fine for a month, what's causing the sudden crash? can it be fixed without increasing server size?"

Thanks for your answers 🙂

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 13, 2021, 12:45pm UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/4 "2021-10-13T12:45:59Z")

</div>

Bonjour Cyril 🙂

> [@cyril.r](#):
>
> it is working fine for a month, what's causing the sudden crash?

Well. The number of indices increased so yes, even if you don't query them, they are taking a significant part of the memory, specifically in the cluster state.  
If you don't want to delete the non needed indices, then you should close the ones you are not using.

> can it be fixed without increasing server size?

Yes. More likely it will be fine again if you remove the non needed indices.  
You should also disable monitoring or index the monitoring data in a separated cluster not in the production cluster itself.

What is the output of:

```auto
GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

```

If some outputs are too big, please share them on [gist.github.com](http://gist.github.com) and link them here.

---

<div class="post-metadata">

**Author:** ![cyril.r](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@cyril.r](https://discuss.elastic.co/u/cyril.r)\
**Post date:** [October 13, 2021, 1:00pm UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/5 "2021-10-13T13:00:34Z")

</div>

Unfortunately I can't close any indices since they could be needed at anytime by our application.

I'm having a hard time to make 100% sure that the crashes are bound to the oversharding and not anything else.

Here are the outputs you requested:

GET / = [GET / · GitHub](https://gist.github.com/cyril-rm/bf859705d7f6a5011ff74e5dc72d5bcf)  
GET /\_cat/nodes?v = [/\_cat/nodes?v · GitHub](https://gist.github.com/cyril-rm/8ef1397f958896b0b1b78684cfd2f923)  
GET /\_cat/health?v = [/\_cat/health?v · GitHub](https://gist.github.com/cyril-rm/f666f9683bf262e032bb94a335c3bdd3)  
GET /\_cat/indices?v = [GET /\_cat/indices?v · GitHub](https://gist.github.com/cyril-rm/64408f42cf9072918b898c4d1de26ae8)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 13, 2021, 1:03pm UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/6 "2021-10-13T13:03:24Z")

</div>

Could you share the non redacted index names for all indices which are not the ones you are using yourself? Update your gist with that information please.

---

<div class="post-metadata">

**Author:** ![cyril.r](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@cyril.r](https://discuss.elastic.co/u/cyril.r)\
**Post date:** [October 13, 2021, 1:15pm UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/7 "2021-10-13T13:15:04Z")

</div>

I updated the gist and left the names of indices that our app doesnt use

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 13, 2021, 1:46pm UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/8 "2021-10-13T13:46:05Z")

</div>

You have a lot of indices with 5 or 2 shards . Just use one shard. Unless you have more than 20gb of data, one shard is enough.

As you have only one node, you should set the number of replicas to 0. It does not make sense to ask for replicas if you don't have nodes to replicate the data. That won't change a lot of things but that will just clean up a bit the things.

You have a lot of indices for very few documents. Is there a reason for this? I can't judge by the name (because it's redacted) if they are time based indices or not.  
Could you tell more about the use case or why do you have so many tiny indices?

---

<div class="post-metadata">

**Author:** ![cyril.r](https://avatars.discourse-cdn.com/v4/letter/c/96bed5/32.png) [@cyril.r](https://discuss.elastic.co/u/cyril.r)\
**Post date:** [October 13, 2021, 4:14pm UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/9 "2021-10-13T16:14:48Z")

</div>

I don't have full visibility of what the devs are doing,

I think that for a particular feature they create an indice per customer due so some constraints, so the number of indices will increase over time but they won't get the same amount of documents (it depends of the usage).

The indices are unrelated (no rollover/time-based) and handled (created) by the application.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 14, 2021, 1:05pm UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/10 "2021-10-14T13:05:28Z")

</div>

> [@cyril.r](#):
>
> I don't have full visibility of what the devs are doing,

You can't expect miracles if you don't have any control with so few ressources.

> [@cyril.r](#):
>
> I think that for a particular feature they create an indice per customer

That could be a bad idea specifically for small datasets and small heaps.  
Using filtered aliases might be better.

But again, I don't know the use case.

Coming back to one of your original questions:

> [@cyril.r](#):
>
> I have to provide a strong and concise answer to my manager, I understand according to [Size your shards | Elasticsearch Guide [7.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html) that we are oversharding, but the point of my manager is "it is working fine for a month, what's causing the sudden crash? can it be fixed without increasing server size?"

You can't have both:

- no control of what devs are doing (including oversharding)
- no control of the size of RAM/HEAP available

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 11, 2021, 1:05pm UTC](https://discuss.elastic.co/t/out-of-memory-crash-few-documents-load/286597/11 "2021-11-11T13:05:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.