# Storing into Elasticsearch using Apache Pig **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [September 21, 2016, 3:21pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146 "2016-09-21T15:21:15Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 21, 2016, 3:21pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/1 "2016-09-21T15:21:15Z")
![](https://us1.discourse-cdn.com/elastic/original/2X/f/f78b513166105489633f5caa9c0405ee62a37ccf.PNG) **Hello I have the following Json input in second\_mapping\_data.json :** {"address":"1.1.1.1","logName":"-","user":"-","time":"07/Mar/2014:16:05:49 -0800","method":"GET","uri":"/twiki/bin/edit/Main/Double\_bounce\_sender?topicparent=Main.ConfigurationVariables","protocol":"HTTP/1.1","status":"401","bytes":"1000"} **My pig script :** REGISTER elasticsearch-hadoop-2.0.2.jar REGISTER elasticsearch-hadoop-2.0.2.jar DEFINE EsStorage org.elasticsearch.hadoop.pig.EsStorage(); logs = load 'second\_mapping\_data.json' using PigStorage('\n'); STORE logs INTO 'test\_index/logsdetails' USING org.elasticsearch.hadoop.pig.EsStorage('es.nodes=127.0.0.1:9200'); **and it's not storing the result into elasticsearch and showing error :** Please Find the attachment for the error . Can anyone tell me what is the problem ? ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [September 21, 2016, 4:37pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/2 "2016-09-21T16:37:04Z")
@cloudxy It looks like the client is breaking while discovering the nodes to write to. Could you share which version of Elasticsearch you are using? ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 21, 2016, 4:55pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/5 "2016-09-21T16:55:16Z")
2.4.0 ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [September 21, 2016, 5:13pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/6 "2016-09-21T17:13:49Z")
I've narrowed down the issue. This is a problem with some old IP parsing code and should be fixed in ES-Hadoop v2.2.0 and above. Since you are using Elasticsearch version 2.4.0, I would advise upgrading to ES-Hadoop v2.4.0 to ensure maximum compatibility between the two. Hope this helps! ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 22, 2016, 6:19am UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/7 "2016-09-22T06:19:28Z")
Ok . Thank You . I'll try and let you know . ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 22, 2016, 6:31am UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/8 "2016-09-22T06:31:34Z")
It's hitting elasticsearch but data is not inserting . ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 22, 2016, 6:53am UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/9 "2016-09-22T06:53:24Z")
Thank You so much for helping , it is going to elasticsearch now . I only load that using JsonLoader() . ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 22, 2016, 7:59am UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/10 "2016-09-22T07:59:40Z")
**But when I'm trying to do the same thing using aws elasticsearch service the following exception is showing :** d per job phase (AliasName[line,offset]): M: logs[1,7] C: R: 2016-09-22 00:41:14,066 [LocalJobRunner Map Task Executor #0] INFO org.elasticsearch.hadoop.util.Version - Elasticsearch Hadoop v2.4.0 [4ab7a3ea4c] 2016-09-22 00:41:14,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job\_local611031861\_0002 2016-09-22 00:41:14,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases logs 2016-09-22 00:41:14,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: logs[1,7] C: R: 2016-09-22 00:41:14,277 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2016-09-22 00:41:19,986 [communication thread] INFO org.apache.hadoop.mapred.LocalJobRunner - map \> map 2016-09-22 00:42:17,458 [LocalJobRunner Map Task Executor #0] INFO org.apache.commons.httpclient.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection timed out 2016-09-22 00:42:17,458 [LocalJobRunner Map Task Executor #0] INFO org.apache.commons.httpclient.HttpMethodDirector - Retrying request But in my local server it is running fine . Can you please tell me what is the problem ? ![](https://us1.discourse-cdn.com/elastic/original/2X/d/dcb0734455c9675aac575d2b9fb0c6257fe6b0d5.PNG) ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [September 22, 2016, 3:15pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/11 "2016-09-22T15:15:42Z")
@cloudxy As detailed [here](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/cloud.html#_configure_the_connector_to_run_in_wan_mode), when running against a cloud environment, you may need to set the `es.nodes.wan.only` option. This is because ES Hadoop asks one server in the cluster for the addresses of all of the nodes it is aware of. These addresses may not be accessible if Elasticsearch is running in a cloud environment, as they may be behind a firewall or part of a private network. The `es.nodes.wan.only` option tells the connector to not do any node discovery, and to only make connections to the Elasticsearch cluster through the nodes declared in the `es.nodes` setting. ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 22, 2016, 5:50pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/12 "2016-09-22T17:50:45Z")
What I have to write ? store logs INTO 'xx/yy' USING org.elasticsearch.hadoop.pig.EsStorage('es.nodes=the\_endpoint\_of\_the\_node','es.nodes.wan.only=true'); The above one ? ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [September 22, 2016, 6:42pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/13 "2016-09-22T18:42:41Z")
That should do it, yes ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 23, 2016, 5:56am UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/14 "2016-09-23T05:56:51Z")
But still giving problem . I'm trying to push it from my local machine to aws elasticsearch . After the last command when the job is started , it is not ending . **Error:** 2016-09-23 03:08:10,606 [Thread-11] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent 2016-09-23 03:08:10,606 [Thread-11] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2016-09-23 03:08:10,606 [Thread-11] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2016-09-23 03:08:10,606 [Thread-11] INFO org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter is org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter 2016-09-23 03:08:10,609 [Thread-11] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for 2016-09-23 03:08:10,852 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases logs 2016-09-23 03:08:10,852 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: logs[2,7] C: R: 2016-09-23 03:08:10,854 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete java.lang.Exception: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:196) at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:379) at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173) at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:149) at org.elasticsearch.hadoop.pig.EsStorage.putNext(EsStorage.java:192) Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[[search-beyblade-nsmryapnku2giqicww67slo3zi.us-east-1.es.amazonaws.com:9200](http://search-beyblade-nsmryapnku2giqicww67slo3zi.us-east-1.es.amazonaws.com:9200)]] HadoopVersion PigVersion UserId StartedAt FinishedAt Features 2.6.0-cdh5.5.0 0.12.0-cdh5.5.0 root 2016-09-23 03:08:10 2016-09-23 03:12:23 UNKNOWN Failed! Failed Jobs: JobId Alias Feature Message Outputs job\_local685037827\_0002 logs MAP\_ONLY Message: Job failed! test\_index/logsdtls, Input(s): Failed to read data from "file:///home/cloudera/Desktop/Satish/json/second\_mapping\_data.json" Output(s): Failed to produce result in "test\_index/logsdtls" Job DAG: job\_local685037827\_0002 2016-09-23 03:12:23,607 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed! grunt\> 2016-09-23 03:12:28,648 [communication thread] INFO org.apache.hadoop.mapred.LocalJobRunner - map \> map The same error like before is coming . ![](https://us1.discourse-cdn.com/elastic/original/2X/7/7b82e915ee43273a4fa45fe6773cdf1018de41be.png) ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 23, 2016, 1:03pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/15 "2016-09-23T13:03:14Z")
I think it is unable to connect my elasticsearch cluster . Is it the fault of the jar or I have to make changes in my code ? ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [September 23, 2016, 5:23pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/16 "2016-09-23T17:23:22Z")
Normally this occurs when the URL provided is not a valid URL to be used from outside AWS. Generally, if you can query the URL using `curl` locally, it should work correctly with ES-Hadoop if you're running hadoop/pig locally. If the URL does not return, then you will need to provide some other form of connecting to the remote service. This is usually different for every cloud provider. ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 23, 2016, 5:36pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/17 "2016-09-23T17:36:10Z")
I tried making index from local machine using curl command and it worked . ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [September 23, 2016, 6:31pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/18 "2016-09-23T18:31:39Z")
I'm sorry, but I'm afraid I don't have an answer for the library not connecting in this instance. As I said before, different cloud providers have different ways of interacting with Elasticsearch, and everyone's networking situation is diverse. I must also disclose that Elastic does not support the AWS Elasticsearch Service. My last bit of advice would be to increase the logging levels to `TRACE` and check the output for network related exceptions that may be getting swallowed and logged. Hopefully this will shed some light on the connectivity problems. ---
**Author:** ![cloudxy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cloudxy/32/86056_2.png) [@cloudxy](https://discuss.elastic.co/u/cloudxy)\ **Post date:** [September 27, 2016, 12:09pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/19 "2016-09-27T12:09:14Z")
Data is going to elasticsearch service from cloudera just have to mention the es.port=80 . 🙂 . ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:23pm UTC](https://discuss.elastic.co/t/storing-into-elasticsearch-using-apache-pig/61146/20 "2017-07-06T13:23:05Z")