# Data Loss in elasticsearch in high load **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2018, 11:48am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515 "2018-01-08T11:48:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 8, 2018, 11:48am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/1 "2018-01-08T11:48:07Z")
Hi All, We are facing a strange and critical issue while testing elastic-search in high speed transaction monitoring. We are using Elastic-search for dumping high speed data coming from JMX MXbean. Our JMX listeners are getting objects from MXbean and using Elastic search High Level Java REST API, we are sending these objects (as JSON format) for indexing. Note : We are using Bulk insert with 500 document at once. With single JMX object, its working good, But when we start load tests, first few iterations for 2 min (with about 150K documents) goes fine, and after sometime we will see Listeners are receiving more data but Document inserted in elastic-search are quite less. For example, I sent 150K Documents, but at the end of Load test, the document count in elastic search shows only 148K. There is no error i see in logs. Any one faced this issue? How can we resolve this? I saw, good design will be to dump into DB first and then from there using Log-stash we can get or index those documents. Is this only option left with me? Please suggest. Thanks, Param ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 8, 2018, 11:54am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/2 "2018-01-08T11:54:29Z")
Are you checking the bulk request response for errors and retrying documents that may have failed? How large is your Elasticsearch cluster? ---
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 9, 2018, 5:13am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/4 "2018-01-09T05:13:30Z")
Yes, I am checking the bulk response status. There is no error. I do not have any cluster. Its one node, Development machine. Thanks, Param ---
**Author:** ![Abhilash\_Bolla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_bolla/32/26840_2.png) [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)\ **Post date:** [January 9, 2018, 5:42am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/5 "2018-01-09T05:42:04Z")
Are you checking for documents which have failed getting indexed, because of maybe mapping issues, parsing exceptions, etc. Check logs for the same. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 9, 2018, 6:40am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/6 "2018-01-09T06:40:37Z")
Is it possible that some of the documents you're sending have the same id? For instance, if you use the current timestamp in millis and are generating more than one document in the same milisecond, that might happen. ---
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 9, 2018, 9:01am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/7 "2018-01-09T09:01:08Z")
Thanks all for response. I got little deep into bulk response and seeing this error. Failure message : [service\_transaction\_1/0bFoO7ZIQGuFCpdGT\_KvZA][[service\_transaction\_1][4]] ElasticsearchException[Elasticsearch exception [type=version\_conflict\_engine\_exception, reason=[service\_transaction\_type\_1][df5ea2ea-3278-4724-bce2-89e3dbfdf1aa]: version conflict, document already exists (current version [1])]] Is failed : true .... I know now, that this error i am getting, as elasticsearch is trying to create a document of same version which already exists [1]. Question is, why elastic search creating same document, as i am passing ID explicitly, in above example df5ea2ea-3278-4724-bce2-89e3dbfdf1aa. But when i search this document in elastic, it says not found. GET service\_transaction\_1/service\_transaction\_type\_1/df5ea2ea-3278-4724-bce2-89e3dbfdf1aa { "\_index": "service\_transaction\_1", "\_type": "service\_transaction\_type\_1", "\_id": "df5ea2ea-3278-4724-bce2-89e3dbfdf1aa", "found": false } I am inserting document As below by passing explicit ID. IndexRequest indexRequest = new IndexRequest("service\_transaction\_1", "service\_transaction\_type\_1", UUID.randomUUID().toString()) .source(putJsonDocumentTrans(obj.combinedBpelObj, obj.transobj)); indexRequest.opType(DocWriteRequest.OpType.CREATE); requestBulk.add(indexRequest); Is there any way to disable versioning completely for my index ? Thanks, Param ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 9, 2018, 9:09am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/8 "2018-01-09T09:09:57Z")
> [@paramlogicoy](#): > > CREATE Why do you use an opType? If you're generating new IDs for each document, you don't really need that. Version conflicts come from there. Just remove that line. What happens? ---
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 9, 2018, 9:11am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/9 "2018-01-09T09:11:03Z")
> [@val](#): > > Fo Might be the reason, Now i changed the API call by passing my ID explicitly. As mentioned in my today's post. Could you please check, Do you see any issues in my way of inserting doc? Basically i wants only create operation for each of my insert call. But i am getting version issue which i am not getting why!!. As if i am creating a unique id of document from my call, why this version issue should come ☹ ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 9, 2018, 9:11am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/10 "2018-01-09T09:11:43Z")
See my previous comment ---
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 9, 2018, 9:13am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/11 "2018-01-09T09:13:27Z")
> [@val](#): > > Just remove that line. What happens? If i will remove this line, I will see Data loss, As original issue. I added this today just to make sure elasticsearch will do only create not update. Param ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 9, 2018, 9:17am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/12 "2018-01-09T09:17:23Z")
Ok, can you try to remove your UUID generation and let ES generate its own IDs. What happens? ---
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 9, 2018, 9:21am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/13 "2018-01-09T09:21:08Z")
This is what the original design i had. ES was generating its own ID. In this case if i am sending 150K document insert request, i see only 148 / 149K document inserted in elasticsearch. POST service\_transaction\_1/\_count Before every load test, i delete the index. Param ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 9, 2018, 9:30am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/14 "2018-01-09T09:30:00Z")
When getting the document count at the end of the benchmark, do you wait for a final refresh to occur? ---
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 9, 2018, 10:03am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/15 "2018-01-09T10:03:07Z")
Yes, I tried that also. Before my load test executes, i execute PUT bpel\_mon\_event/\_settings { "index" : { "refresh\_interval" : "-1" } } and once the load test is finish i set it back to 1s. Still count is not same as i am sending. Param ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 9, 2018, 10:04am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/16 "2018-01-09T10:04:34Z")
The index is called `service_transaction_1` but you're modifying the refresh\_interval of `bpel_mon_event`? or is that an alias? ---
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 9, 2018, 10:12am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/17 "2018-01-09T10:12:58Z")
Sorry, service\_transaction\_1. We have two indexes where data is flowing. I pasted query for one of them Big problem is : The behaviour is not consistent. Sometime i do not get any error if i am sending id from my API. Sometime, i gets version issue. If i remove OptType 'CREATE' , I see documents are missing. Should i use LogStash in this case? What will you suggest. But if i use LogStash, it will not be real time data, As i will configure logstash script to run after some interval. Param ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 9, 2018, 10:46am UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/18 "2018-01-09T10:46:31Z")
Unless you need to update documents based on an external ID, I would recommend letting Elasticsearch create the document IDs as this gives better indexing throughput. Once you set the refresh\_rate back to 1, you need to wait a while to ensure that a refresh takes place before retrieving the document count. You may also be able to [force a refresh at the end of the benchmark using the refresh API](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/indices-refresh.html) before you get the document count. ---
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 9, 2018, 12:15pm UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/19 "2018-01-09T12:15:51Z")
> [@paramlogicoy](#): > > Yes, I am checking the bulk response status. There is no error. I do not have any cluster. Its one node, Development machine. I tried this also. But No Luck ☹ .. I am sharing my Program, If anyone can review this and share thoughts, what wrong i am doing. import java.io.IOException; import java.util.HashMap; import java.util.HashSet; import java.util.Map; import java.util.Set; import java.util.UUID; import java.util.logging.Level; import java.util.logging.Logger; import org.apache.http.HttpHost; import org.elasticsearch.action.bulk.BulkRequest; import org.elasticsearch.action.index.IndexRequest; import org.elasticsearch.client.RestClient; import org.elasticsearch.client.RestHighLevelClient; indent preformatted text by 4 spaces /\*\* - @author Param \*/ public class SampleLoadProgram { TO FETCH UNIQUE trans\_id, I am using following GET query (May be something is wrong here) GET service\_transaction/service\_transaction\_type/\_search { "size":"0", "aggs" : { "total\_trans\_id" : { "cardinality" : { "field" : "trans\_id.keyword"} } } } Please help. Param ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 9, 2018, 12:17pm UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/20 "2018-01-09T12:17:49Z")
With the `cardinality` aggregation, beware that the [count is approximate](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html#_counts_are_approximate) ---
**Author:** ![paramlogicoy](https://avatars.discourse-cdn.com/v4/letter/p/e5b9ba/32.png) [@paramlogicoy](https://discuss.elastic.co/u/paramlogicoy)\ **Post date:** [January 9, 2018, 12:25pm UTC](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515/21 "2018-01-09T12:25:11Z")
OHH.. Thanks. What is the correct way to fetch unique ids in my case? Could you please guide? It seems the document count is coming correct after explicit refresh POST POST /service\_transaction/\_refresh But My GET query is not showing correct count, As you said it is approximate. But for some reason we need exact count of unique transactions. This is response now. { "took": 0, "timed\_out": false, "\_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 50000, "max\_score": 0, "hits": [] }, "aggregations": { "total\_trans\_id": { "value": 9962 } } } Hits \> Total is document count : Perfact count. But total trans\_id should be 10000, but it is 9962 Param [Next page](https://discuss.elastic.co/t/data-loss-in-elasticsearch-in-high-load/114515.md?page=2)