# Customer reports documents 'disappearing' from 2 node index?! **URL:** \ **Category:** Elasticsearch\ **Created:** [August 21, 2012, 1:54pm UTC](https://discuss.elastic.co/t/customer-reports-documents-disappearing-from-2-node-index/8800 "2012-08-21T13:54:05Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Charlie\_Ott](https://avatars.discourse-cdn.com/v4/letter/c/d78d45/32.png) [@Charlie\_Ott](https://discuss.elastic.co/u/Charlie_Ott)\ **Post date:** [August 21, 2012, 1:54pm UTC](https://discuss.elastic.co/t/customer-reports-documents-disappearing-from-2-node-index/8800/1 "2012-08-21T13:54:05Z")
Running Windows Server 2008 R2 with 2 nodes @ version 0.19.1 (JRE6u32). We have only been given 2GB of memory for this virtual server (Which is ridiculous, i know). We are now at about 270k documents and the customer has reported that since Friday our record counts have dropped without user intervention. Everything is stored in a single index and seperated on our web application interface by "provider name". One provider set had 48 records, now reporting 36. Another has just been reported to be missing 95 records. So i know it is not specific to a dataset, but more specific to the index as a whole. I am curious as to how many documents our search nodes can hold, and how JVM heap space contributes. Additionally this would be scaled differently based on document size, no? We are always getting more data (10-50k monthly) and I am having trouble keeping elasticsearch stable as demand for document storage grows. I am at the point now where I am willing to push for paid support, but not sure if that is entirely neccesary at this point. We are currently checking the back end RDBMS that maintains all of the original documents and user data for activity that might lead to the record loss, but it is looking like there was no user intervention that caused these documents to disappear. We have been running ES for about a year and a half now and have not seen this before. Any ideas? Thanks in advance, Charles -- ---
**Author:** ![Charlie\_Ott](https://avatars.discourse-cdn.com/v4/letter/c/d78d45/32.png) [@Charlie\_Ott](https://discuss.elastic.co/u/Charlie_Ott)\ **Post date:** [August 21, 2012, 1:56pm UTC](https://discuss.elastic.co/t/customer-reports-documents-disappearing-from-2-node-index/8800/2 "2012-08-21T13:56:53Z")
default-mapping.json: { "_default_": { "dynamic\_templates": [{ "index\_nonanalyzed": { "match": "\*", "mapping": { "index": "not\_analyzed" } } }] } } -- ---
**Author:** ![Charlie\_Ott](https://avatars.discourse-cdn.com/v4/letter/c/d78d45/32.png) [@Charlie\_Ott](https://discuss.elastic.co/u/Charlie_Ott)\ **Post date:** [August 21, 2012, 2:51pm UTC](https://discuss.elastic.co/t/customer-reports-documents-disappearing-from-2-node-index/8800/3 "2012-08-21T14:51:14Z")
Log files on the production system from Aug-17 reported JVM was out of memory. We are using the Tanuki service wrapper, so I had the production support staff look at the maximum memory limit in the JVM wrapper configuration. It was commented out (#). I assume this allowed the JVM to grow indefinitely, or until the OS constrained. The RDBMS showed all records were in fact indexed at some point, so I assumed they were still in the /data folder and had perhaps been lost in RAM. After setting the JVM limits to 512MB, and restarting the nodes all records are back. (phew!) We do not have mlock enabled, and I am not sure how efficient mlock would be running in VMware anyway. Moving forward if anyone has recommendations on managing growth better I would like to hear your comments. Thanks, Charles -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:15am UTC](https://discuss.elastic.co/t/customer-reports-documents-disappearing-from-2-node-index/8800/4 "2017-07-06T03:15:40Z")