# \[hadoop\] Extra Documents in Elastic Search **URL:** \ **Category:** Elasticsearch\ **Created:** [April 23, 2014, 9:15pm UTC](https://discuss.elastic.co/t/hadoop-extra-documents-in-elastic-search/17156 "2014-04-23T21:15:19Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Napoleon\_T](https://avatars.discourse-cdn.com/v4/letter/n/e5b9ba/32.png) [@Napoleon\_T](https://discuss.elastic.co/u/Napoleon_T)\ **Post date:** [April 23, 2014, 9:15pm UTC](https://discuss.elastic.co/t/hadoop-extra-documents-in-elastic-search/17156/1 "2014-04-23T21:15:19Z")
Hi, I'm trying to store a lot of documents into ES using pig. The pig job ends successfully but I end up with more documents in Elasticsearch than the number of rows in my input. My pig script is 3 lines: REGISTER 'local/path/to/m2.jar' data = load 'path/to/hdfs/file.tsv' as (field1: chararray, field2: long, field3: long, field4: long) store data into 'index/type' using org.elasticsearch.hadoop.pig.EsStorage('[es.nodes=node2.domain.com](http://es.nodes=node2.domain.com)', 'es.rersource=index/type'); I have speculative execution disabled for map and reduce when running this pig script. Hadoop states that 54,723,557 records were written (console output and job tracker UI). ES head plugin claims that I have docs: 57,344,987 (57,344,987). My environment: hadoop: 1.2.1 with 6 nodes cluster elasticsearch: 1.0.0. 6 node cluster. Different than hadoop nodes. elasticsearch-hadoop version M2. Pig version: 0.12.0 Any ideas of what is going on here? Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a852acc3-c331-491b-817d-4386493aec90%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a852acc3-c331-491b-817d-4386493aec90%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Napoleon\_T](https://avatars.discourse-cdn.com/v4/letter/n/e5b9ba/32.png) [@Napoleon\_T](https://discuss.elastic.co/u/Napoleon_T)\ **Post date:** [May 15, 2014, 7:37pm UTC](https://discuss.elastic.co/t/hadoop-extra-documents-in-elastic-search/17156/2 "2014-05-15T19:37:38Z")
I did not entirely solve this issue. But it looks like ES is dropping some requests when it's overloaded. As my hadoop cluster can handle 42 mappers, I had 42 tasks trying to send write requests to only 1 ES node (I believe all the requests only go to one node is ES). Most of the time, many tasks will fail and my hadoop job will fail. But sometimes, hadoop returns a success and not all the data has been successfully written. Reducing the number of mappers should have helped, but for some reasons running pig with the property -Dmapred.tasktracker.map.tasks.maximum=1 did not do the trick. Limiting the number of mappers directly in the cluster conf files seems to have solved the problem. On Wednesday, April 23, 2014 4:15:19 PM UTC-5, Napoleon T. wrote: > Hi, > > I'm trying to store a lot of documents into ES using pig. The pig job ends > successfully but I end up with more documents in Elasticsearch than the > number of rows in my input. > My pig script is 3 lines: > REGISTER 'local/path/to/m2.jar' > data = load 'path/to/hdfs/file.tsv' as (field1: chararray, field2: long, > field3: long, field4: long) > store data into 'index/type' using > org.elasticsearch.hadoop.pig.EsStorage('[es.nodes=node2.domain.com](http://es.nodes=node2.domain.com)', > 'es.rersource=index/type'); > > I have speculative execution disabled for map and reduce when running this > pig script. > > Hadoop states that 54,723,557 records were written (console output and > job tracker UI). > ES head plugin claims that I have docs: 57,344,987 (57,344,987). > > My environment: > hadoop: 1.2.1 with 6 nodes cluster > elasticsearch: 1.0.0. 6 node cluster. Different than hadoop nodes. > elasticsearch-hadoop version M2. > Pig version: 0.12.0 > > Any ideas of what is going on here? > > Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b819e576-f0ef-41d4-854a-63bab811951a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b819e576-f0ef-41d4-854a-63bab811951a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [May 16, 2014, 8:50am UTC](https://discuss.elastic.co/t/hadoop-extra-documents-in-elastic-search/17156/3 "2014-05-16T08:50:38Z")
Hi, I strongly recommend using the latest release (es-hadoop 2.0 RC1) which handles document rejections (which can and will happen when ES is overloaded). Simply replace the jar and start adding more tasks until you get the desired performance. Know that es-hadoop also records stats about the job (docs rejected, sent, accepted, etc...) which are displayed at the end of the job so you can use that information as well to double-check the number of docs added. Cheers, On Thu, May 15, 2014 at 10:37 PM, Napoleon T. [napoleon13e@gmail.com](mailto:napoleon13e@gmail.com) wrote: > I did not entirely solve this issue. But it looks like ES is dropping some > requests when it's overloaded. As my hadoop cluster can handle 42 mappers, > I had 42 tasks trying to send write requests to only 1 ES node (I believe > all the requests only go to one node is ES). Most of the time, many tasks > will fail and my hadoop job will fail. But sometimes, hadoop returns a > success and not all the data has been successfully written. > Reducing the number of mappers should have helped, but for some reasons > running pig with the property -Dmapred.tasktracker.map.tasks.maximum=1 did > not do the trick. > Limiting the number of mappers directly in the cluster conf files seems to > have solved the problem. > > On Wednesday, April 23, 2014 4:15:19 PM UTC-5, Napoleon T. wrote: > > > Hi, > > > > I'm trying to store a lot of documents into ES using pig. The pig job > > ends successfully but I end up with more documents in Elasticsearch than > > the number of rows in my input. > > My pig script is 3 lines: > > REGISTER 'local/path/to/m2.jar' > > data = load 'path/to/hdfs/file.tsv' as (field1: chararray, field2: long, > > field3: long, field4: long) > > store data into 'index/type' using org.elasticsearch.hadoop.pig. > > EsStorage('[es.nodes=node2.domain.com](http://es.nodes=node2.domain.com)', 'es.rersource=index/type'); > > > > I have speculative execution disabled for map and reduce when running > > this pig script. > > > > Hadoop states that 54,723,557 records were written (console output and > > job tracker UI). > > ES head plugin claims that I have docs: 57,344,987 (57,344,987). > > > > My environment: > > hadoop: 1.2.1 with 6 nodes cluster > > elasticsearch: 1.0.0. 6 node cluster. Different than hadoop nodes. > > elasticsearch-hadoop version M2. > > Pig version: 0.12.0 > > > > Any ideas of what is going on here? > > > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b819e576-f0ef-41d4-854a-63bab811951a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b819e576-f0ef-41d4-854a-63bab811951a%40googlegroups.com) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmc5NOAmsjhVca2c%2B%3D5pCdHPPfyZPDPgagD5PWM3%3Da09Pg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmc5NOAmsjhVca2c%2B%3D5pCdHPPfyZPDPgagD5PWM3%3Da09Pg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:29am UTC](https://discuss.elastic.co/t/hadoop-extra-documents-in-elastic-search/17156/4 "2017-07-06T01:29:00Z")