:9300", "NONE"],
"attributes" : {
"max_local_storage_nodes" : "1",
"master" : "true"
},
"os" : {
"timestamp" : 1675868570502,
"cpu_percent" : 2,
"load_average" : 0.02,
"mem" : {
"total_in_bytes" : 32153341952,
"free_in_bytes" : 306769920,
"used_in_bytes" : 31846572032,
"free_percent" : 1,
"used_percent" : 99
},
"swap" : {
"total_in_bytes" : 0,
"free_in_bytes" : 0,
"used_in_bytes" : 0
}
}
}
}
}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 8, 2023, 4:17pm UTC](https://discuss.elastic.co/t/heap-memory-full-new-documents-just-disappeared/325037/3 "2023-02-08T16:17:12Z")
> [@Pete\_Watcharawit1](#):
>
> Our cluster is using ES version 2.3.3
This is very, very old and I have not used it in years. You should really look to upgrade.
How are your nodes configured? Have you `minimum_master_nodes` correctly set (`2` if you have 2 or 3 nodes in the cluster).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 8, 2023, 9:22pm UTC](https://discuss.elastic.co/t/heap-memory-full-new-documents-just-disappeared/325037/4 "2023-02-08T21:22:18Z")
Welcome to our community! 😃
> [@Pete\_Watcharawit1](#):
>
> Our cluster is using ES version 2.3.3
This is positively ancient and no longer supported, you need to upgrade as a matter of high urgency.
---
### Author:  [@Pete\_Watcharawit1](https://discuss.elastic.co/u/Pete_Watcharawit1)
#### Post date: [February 10, 2023, 4:11am UTC](https://discuss.elastic.co/t/heap-memory-full-new-documents-just-disappeared/325037/5 "2023-02-10T04:11:33Z")
We have 2 nodes and they are both data and master. My question: os.mem.used\_percent = 99% is this dangerous? What parameter should be added for config/elasticsearch.yml file to increase this?
Our two servers have 1.5 TB storage for each node and I am sure we can increase the storage for the nodes.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 10, 2023, 6:07am UTC](https://discuss.elastic.co/t/heap-memory-full-new-documents-just-disappeared/325037/6 "2023-02-10T06:07:59Z")
What have you got `minimum_master_nodes` set to? If this is set to `1` and you have 2 nodes, [the cluster is incorrectly configured and can suffer from split brain](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-node.html#split-brain) if there is an issue with any of the nodes. This can often cause data loss like you describe.
Note that you can not have a highly available cluster with only 2 nodes. Elasticsearch require a strict majority of master eligible nodes to elect a master, so in a correctly configured cluster you would not be able to elect a master if one of two nodes went down and the cluster would no longer accept writes. If you however have 3 master eligible nodes in the cluster you can lose one node and still allow the remaining 2 to form a majority and elect a master, which will allow the cluster to continue operating.
OS memory is often fully used when using Elasticsearch as the page cache will get filled up, which is not an issue and perfectly normal.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 10, 2023, 6:08am UTC](https://discuss.elastic.co/t/heap-memory-full-new-documents-just-disappeared/325037/7 "2023-03-10T06:08:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.