# Elasticsearch Data Loss in Index in ELK 6.4.2 **URL:** \ **Category:** Elasticsearch\ **Created:** [May 22, 2020, 8:34am UTC](https://discuss.elastic.co/t/elasticsearch-data-loss-in-index-in-elk-6-4-2/233866 "2020-05-22T08:34:37Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Shreyash](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@Shreyash](https://discuss.elastic.co/u/Shreyash)\ **Post date:** [May 22, 2020, 8:34am UTC](https://discuss.elastic.co/t/elasticsearch-data-loss-in-index-in-elk-6-4-2/233866/1 "2020-05-22T08:34:37Z")
Hi All, I am using ELK stack 6.4.2 with Microsoft SQL Server. The Elasticsearch, Logstash are running in a Windows Virtual Machine. We have observed that a randomw document which was already index before goes missing from an Elasticsearch index. We are restoring the document again by changing a date column and pushing it in index through Logstash. Even after restoring the document in index, the document goes missing in the index again sometimes. This behavior is unpredictable. We have observed this behavior recently. Please find below extra information regarding our environment: - We have around 5 GB data in the Elasticsearch node. - We are using a single node structure. - The Elasticsearch cluster, inedex health is yellow all the time. - We have replicas for every index as well. - We have multiple instances of Logstash running at a time on each index - Elasticsearch showed an error - _low disk watermark [85%]_ , hence we kept more than 85% free space in the directory. But this issue is still observed. Expecting a quick solution or reply. Thanks in advance. Best Regards, Shreyash ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 24, 2020, 11:37am UTC](https://discuss.elastic.co/t/elasticsearch-data-loss-in-index-in-elk-6-4-2/233866/2 "2020-05-24T11:37:02Z")
> [@Shreyash](#): > > - The Elasticsearch cluster, inedex health is yellow all the time. > - We have replicas for every index as well. Elasticsearch will never deploy a replica shard to the same node as where the primary resides so having a replica configured when you only have one node does not make any sense and will always lead to yellow indices. > [@Shreyash](#): > > Elasticsearch showed an error - _low disk watermark [85%]_ , hence we kept more than 85% free space in the directory. But this issue is still observed. What is the output of the `_cluster/stats` API? What does your Logstash config look like? How do you identify that the document has gone missing? ---
**Author:** ![Shreyash](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@Shreyash](https://discuss.elastic.co/u/Shreyash)\ **Post date:** [May 24, 2020, 5:28pm UTC](https://discuss.elastic.co/t/elasticsearch-data-loss-in-index-in-elk-6-4-2/233866/3 "2020-05-24T17:28:49Z")
Hi @Christian_Dahlqvist, Thanks for reply. I have observed that the document with a particular "\_id" property goes missing from the index after some time. I have observed this with search queries to the elasticsearch. This might happen during the update of the document by the Logstash. I have also observed that this issue happens with the document having large JSON structure. Logstash sometimes is not able to read the complete JSON string from the column data in the sql server which is of type nvarchar(max). **The Logstash config file looks like this:** ```auto input{ jdbc { jdbc_connection_string => "sqlserverconnection" jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc_user => "user" schedule => "*/2 * * * * *" statement =>"select ID,data from data(Nolock) where ModifiedDate > :sql_last_value order by ModifiedDate" last_run_metadata_path => "C:/logstash-6.4.2/logstash-6.4.2/data/lastRun/.logstash_jdbc_last_run" type=>"data" } } filter{ json { source => "data" target => "Data" remove_field => ["data"] } } output{ if [type]=="data" { elasticsearch { hosts => "localhost:9200" index => "idx_data" document_id => "%{id}" } } stdout { codec => rubydebug } } ``` **Please find the output of the "\_cluster/stats" below**. ```auto { "_nodes": { "total": 1, "successful": 1, "failed": 0 }, "cluster_name": "elasticsearch", "timestamp": 1590339685409, "status": "yellow", "indices": { "count": 25, "shards": { "total": 125, "primaries": 125, "replication": 0.0, "index": { "shards": { "min": 5, "max": 5, "avg": 5.0 }, "primaries": { "min": 5, "max": 5, "avg": 5.0 }, "replication": { "min": 0.0, "max": 0.0, "avg": 0.0 } } }, "docs": { "count": 2357855, "deleted": 3197 }, "store": { "size_in_bytes": 1919953025 }, "fielddata": { "memory_size_in_bytes": 0, "evictions": 0 }, "query_cache": { "memory_size_in_bytes": 713538, "total_count": 152469, "hit_count": 7201, "miss_count": 145268, "cache_size": 609, "cache_count": 2925, "evictions": 2316 }, "completion": { "size_in_bytes": 0 }, "segments": { "count": 607, "memory_in_bytes": 13432603, "terms_memory_in_bytes": 11065404, "stored_fields_memory_in_bytes": 601544, "term_vectors_memory_in_bytes": 0, "norms_memory_in_bytes": 781440, "points_memory_in_bytes": 125299, "doc_values_memory_in_bytes": 858916, "index_writer_memory_in_bytes": 0, "version_map_memory_in_bytes": 0, "fixed_bit_set_memory_in_bytes": 0, "max_unsafe_auto_id_timestamp": -1, "file_sizes": {} } }, "nodes": { "count": { "total": 1, "data": 1, "coordinating_only": 0, "master": 1, "ingest": 1 }, "versions": [ "6.4.2" ], "os": { "available_processors": 16, "allocated_processors": 16, "names": [ { "name": "Windows Server 2016", "count": 1 } ], "mem": { "total_in_bytes": 34359267328, "free_in_bytes": 12594618368, "used_in_bytes": 21764648960, "free_percent": 37, "used_percent": 63 } }, "process": { "cpu": { "percent": 3 }, "open_file_descriptors": { "min": -1, "max": -1, "avg": 0 } }, "jvm": { "max_uptime_in_millis": 303448982, "versions": [ { "version": "1.8.0_191", "vm_name": "Java HotSpot(TM) 64-Bit Server VM", "vm_version": "25.191-b12", "vm_vendor": "Oracle Corporation", "count": 1 } ], "mem": { "heap_used_in_bytes": 704804856, "heap_max_in_bytes": 1037959168 }, "threads": 168 }, "fs": { "total_in_bytes": 243369242624, "free_in_bytes": 47769489408, "available_in_bytes": 47769489408 }, "plugins": [], "network_types": { "transport_types": { "security4": 1 }, "http_types": { "security4": 1 } } } } ``` _ **Best Regards,** _ _ **Shreyash Karmali** _ ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 24, 2020, 5:31pm UTC](https://discuss.elastic.co/t/elasticsearch-data-loss-in-index-in-elk-6-4-2/233866/4 "2020-05-24T17:31:12Z")
What is the average and maximum size of documents? ---
**Author:** ![Shreyash](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@Shreyash](https://discuss.elastic.co/u/Shreyash)\ **Post date:** [May 24, 2020, 5:54pm UTC](https://discuss.elastic.co/t/elasticsearch-data-loss-in-index-in-elk-6-4-2/233866/5 "2020-05-24T17:54:40Z")
Hi @Christian_Dahlqvist, The average size of the document is around 70kb. The max size of the document would be around 5MB. Best Regards, Shreyash Karmali ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 25, 2020, 12:43am UTC](https://discuss.elastic.co/t/elasticsearch-data-loss-in-index-in-elk-6-4-2/233866/6 "2020-05-25T00:43:10Z")
> [@Shreyash](#): > > Expecting a quick solution or reply. FYI it's pretty presumptive, and quite rude, to expect that. ---
**Author:** ![Shreyash](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@Shreyash](https://discuss.elastic.co/u/Shreyash)\ **Post date:** [May 25, 2020, 5:06am UTC](https://discuss.elastic.co/t/elasticsearch-data-loss-in-index-in-elk-6-4-2/233866/7 "2020-05-25T05:06:40Z")
Hi Mark, I am really sorry. But i did not mean it that way. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 22, 2020, 5:06am UTC](https://discuss.elastic.co/t/elasticsearch-data-loss-in-index-in-elk-6-4-2/233866/8 "2020-06-22T05:06:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.