# ES - Amazon EMR - Pig **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [July 24, 2015, 6:14am UTC](https://discuss.elastic.co/t/es-amazon-emr-pig/26212 "2015-07-24T06:14:13Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![ayan\_guha](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@ayan\_guha](https://discuss.elastic.co/u/ayan_guha)\ **Post date:** [July 24, 2015, 6:14am UTC](https://discuss.elastic.co/t/es-amazon-emr-pig/26212/1 "2015-07-24T06:14:13Z")
Hi I am using Pig to connect to ES for both reading and writing data. [hadoop mapreduce]$ hadoop version Hadoop 2.4.0-amzn-4 Subversion [http://aws.amazon.com/elasticmapreduce/](http://aws.amazon.com/elasticmapreduce/) -r 8aaaf366c7fe8ea6d8c37e76c2cc8caa10e11c06 Compiled by Elastic MapReduce on 2015-04-02T21:19Z Compiled with protoc 2.5.0 From source with checksum 6c725ed23b3ecb95921fe461587fccf This command was run using /home/hadoop/.versions/2.4.0-amzn-4/share/hadoop/common/hadoop-common-2.4.0-amzn-4.jar [hadoop mapreduce]$ pig -i Apache Pig version 0.12.0 (rexported) compiled Jan 24 2015, 01:40:48 [hadoop mapreduce]$ I have written this small script and trying in grunt shell register '/home/hadoop/ayan/elasticsearch-hadoop-pig-2.1.0.jar'; DEFINE EsStorage org.elasticsearch.hadoop.pig.EsStorage('es.nodes=xx.xx.xx.xx','es.port=9200'); a = load '/dem/getShipmentById/' using EsStorage(); b = limit a 3; dump b; It is failing with following error 2015-07-24 05:59:38,525 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt\_1433395069625\_0537\_m\_000000\_3 Info:Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected 2015-07-24 05:59:38,525 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed! I understand it is trying to invoke old map-reduce api but why so and what is the solution? ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [July 25, 2015, 11:43am UTC](https://discuss.elastic.co/t/es-amazon-emr-pig/26212/2 "2015-07-25T11:43:41Z")
Unfortunately a bug sneaked into the 2.1.0 release in the Pig integration. It has been fixed in [master](https://github.com/elastic/elasticsearch-hadoop/issues/499) - we plan to release a 2.1.1 shortly to address this however in the meantime you can use the dev[release] [2](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html#download-dev), in particular 2.1.1.BUILD-SNAPSHOT ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 23, 2016, 10:25am UTC](https://discuss.elastic.co/t/es-amazon-emr-pig/26212/3 "2016-09-23T10:25:14Z")
**My pig script is :** REGISTER elasticsearch-hadoop-2.4.0.jar REGISTER piggybank-0.15.0.jar DEFINE EsStorage org.elasticsearch.hadoop.pig.EsStorage(); logs = load 'second\_mapping\_data.json' using JsonLoader('addr: chararray, logname: chararray, user: chararray, time: chararray, method: chararray, uri: chararray, proto: chararray, status: chararray, bytes: chararray'); STORE logs INTO 'test\_index/logsdetails' USING org.elasticsearch.hadoop.pig.EsStorage('es.nodes=endpoint\_of\_aws\_elasticsearch\_cluster','es.nodes.wan.only=true'); When I'm trying this using my grunt shell data is not going to aws elasticsearch service . Can you please tell me what is that I'm missing ? I'm uploading a snapshot of the error . ![](https://us1.discourse-cdn.com/elastic/original/2X/6/627f43b2ce50d6044c7933133275907de3f598b4.PNG) @costin I have full access to AWS ES . **Error :** **1st part of error is attached in the picture :** 2016-09-23 03:08:10,606 [Thread-11] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent 2016-09-23 03:08:10,606 [Thread-11] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2016-09-23 03:08:10,606 [Thread-11] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2016-09-23 03:08:10,606 [Thread-11] INFO org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter is org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter 2016-09-23 03:08:10,609 [Thread-11] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for * * * 2016-09-23 03:08:10,852 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases logs 2016-09-23 03:08:10,852 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: logs[2,7] C: R: 2016-09-23 03:08:10,854 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete java.lang.Exception: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:196) at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:379) at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173) at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:149) at org.elasticsearch.hadoop.pig.EsStorage.putNext(EsStorage.java:192) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98) Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[[search-beyblade-nsmryapnku2giqicww67slo3zi.us-east-1.es.amazonaws.com:9200](http://search-beyblade-nsmryapnku2giqicww67slo3zi.us-east-1.es.amazonaws.com:9200)]] HadoopVersion PigVersion UserId StartedAt FinishedAt Features 2.6.0-cdh5.5.0 0.12.0-cdh5.5.0 root 2016-09-23 03:08:10 2016-09-23 03:12:23 UNKNOWN Failed! Failed Jobs: JobId Alias Feature Message Outputs job\_local685037827\_0002 logs MAP\_ONLY Message: Job failed! test\_index/logsdtls, Input(s): Failed to read data from "file:///home/cloudera/Desktop/Satish/json/second\_mapping\_data.json" Output(s): Failed to produce result in "test\_index/logsdtls" Job DAG: job\_local685037827\_0002 2016-09-23 03:12:23,607 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed! grunt\> 2016-09-23 03:12:28,648 [communication thread] INFO org.apache.hadoop.mapred.LocalJobRunner - map \> map ![](https://us1.discourse-cdn.com/elastic/original/2X/3/3b0ad43ce50a38bf33009cfb510c2b1105ac34e4.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:23pm UTC](https://discuss.elastic.co/t/es-amazon-emr-pig/26212/4 "2017-07-06T13:23:08Z")