# Pig - Lost documents while storing with EsStorage **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [January 12, 2016, 3:13pm UTC](https://discuss.elastic.co/t/pig-lost-documents-while-storing-with-esstorage/38996 "2016-01-12T15:13:30Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![pierregc](https://avatars.discourse-cdn.com/v4/letter/p/73ab20/32.png) [@pierregc](https://discuss.elastic.co/u/pierregc)\ **Post date:** [January 12, 2016, 3:13pm UTC](https://discuss.elastic.co/t/pig-lost-documents-while-storing-with-esstorage/38996/1 "2016-01-12T15:13:31Z")
Hi, I am using the EsStorage class to store millions of documents from a 40 nodes hadoop cluster to a 3 nodes elastic cluster. It is very convenient, but I found out that I am loosing many documents during this process. By default, 70 reducers are instantiated and about 5% of the documents are lost. I reduced manually the number of reducers to 12, and I'm now loosing less than 1% of docs, but I need to reach 0% 🙂 I tried to change tthe es.batch.size.bytes and es.batch.size.entries parameters and although this changes the number of lost documents, I'm still far from 0%. It seems that Pig connector do not verify if the batch of doc was successfully indexed. If a batch failed, it seems not to be retried. Is there any setting parameter to set that I'm missing ? Thanks for your help 🙂 Pierre ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [January 12, 2016, 3:31pm UTC](https://discuss.elastic.co/t/pig-lost-documents-while-storing-with-esstorage/38996/2 "2016-01-12T15:31:50Z")
> [@pierregc](#): > > It seems that Pig connector do not verify if the batch of doc was successfully indexed. If a batch failed, it seems not to be retried. Actually it does. For _every_ batch written, ES-Hadoop will parse the response and, in case documents are rejected, will retry them (just the rejected documents) - this is configurable but out of the box, up to 3 times with 10s wait in between. If that fails, the job fails as well. Does the job fail in your case or not? ES-Hadoop exposes [metrics](https://www.elastic.co/guide/en/elasticsearch/hadoop/master/metrics.html) so you can see at the end of the job, the stats for it. In addition through Marvel you can double check these in real-time. What version of ES and ES-Hadoop are you using? ---
**Author:** ![pierregc](https://avatars.discourse-cdn.com/v4/letter/p/73ab20/32.png) [@pierregc](https://discuss.elastic.co/u/pierregc)\ **Post date:** [January 13, 2016, 10:16am UTC](https://discuss.elastic.co/t/pig-lost-documents-while-storing-with-esstorage/38996/3 "2016-01-13T10:16:35Z")
Thanks for your reply. No the job does not fail. I didn't know about the metrics, that's interesting. I found them and I can see they are actually some Bulk retries. And the number of Documents Accepted is corresponding to 100% of the documents that I want to index. I did a new indexation and some new queries. I found a reason that is explaining most of the losses : I have some documents that have the same key. In that case, only one document is kept in ES. Good ! I'd like to do another test with more reducers but not more time unfortunately. It's good enough for now 🙂 For info : ES version : 1.7.3 ES-Hadoop version : 2.2.0-beta1 (we recently upgraded it because we'll soon upgrade ES to 2.0) Thanks ! ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [January 13, 2016, 10:28am UTC](https://discuss.elastic.co/t/pig-lost-documents-while-storing-with-esstorage/38996/4 "2016-01-13T10:28:50Z")
Glad to hear it's sorted out. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:26pm UTC](https://discuss.elastic.co/t/pig-lost-documents-while-storing-with-esstorage/38996/5 "2017-07-06T13:26:32Z")