# Unable to write data to elasticsearch using hadoop PIG
**URL:** https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157
**Category:** Elasticsearch
**Created:** [March 5, 2014, 5:32am UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157 "2014-03-05T05:32:55Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@siva\_mannem](https://discuss.elastic.co/u/siva_mannem)
#### Post date: [March 5, 2014, 5:32am UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/1 "2014-03-05T05:32:55Z")
I installed ES(at the location /usr/lib/elasticsearch/) on our gateway
server and i am able to run some basic curl commands like XPUT and XGET to
create some indices and retrieve the data in them.
i am able to give single line JSON record but i am unable to give JSON file
as input to curl XPUT .
can anybody give me the syntax for giving JSON file as input for curl XPUT
command?
my next issue is i copied the following 4 elasticsearch-hadoop jar files
elasticsearch-hadoop-1.3.0.M2.jar
elasticsearch-hadoop-1.3.0.M2-sources.jar
elasticsearch-hadoop-1.3.0.M2-javadoc.jar
elasticsearch-hadoop-1.3.0.M2-yarn.jar
to /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
and /usr/lib/gphd/pig/
i have the following json file j.json
++++++
{"k1":"v1" , "k2":"v2" , "k3":"v3"}
++++++++
in my\_hdfs\_path.
my pig script is write\_data\_to\_es.pig
+++++++++++++
REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
A = LOAD '/my\_hdfs\_path/j.json' using
JsonLoader('k1:chararray,k2:chararray,k3:chararray');
STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
++++++++++++++
when i run my pig script
+++++++++
pig -x mapreduce write\_data\_to\_es.pig
++++++++++++
i am getting following error
+++++++++
Input(s):
Failed to read data from "/my\_hdfs\_path/j.json"
Output(s):
Failed to produce result in "usa/ca"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job\_1390436301987\_0089
2014-03-05 00:26:50,839 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser
- ERROR 2997: Input(s):
Failed to read data from "/elastic\_search/es\_hadoop\_test.json"
Output(s):
Failed to produce result in "mannem/siva"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job\_1390436301987\_0089
2014-03-05 00:26:50,839 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser
- _ERROR 2997: Encountered IOException. Out of nodes and retries; caught
exception_
Details at logfile:
/usr/lib/elasticsearch/elasticsearch-0.90.9/pig\_1393997175206.log
++++++++++++
i am using pivotal hadoop version (1.0.1) which is basically apache hadoop
(hadoop-2.0.2)
and pig version is 0.10.1
and elastic search version is 0.90.9
can anybody help me out here?
thank you so much in advance for your help.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f4740e2a-868f-489c-8f9d-842c08ecddff%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f4740e2a-868f-489c-8f9d-842c08ecddff%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@siva\_mannem](https://discuss.elastic.co/u/siva_mannem)
#### Post date: [March 5, 2014, 5:34am UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/2 "2014-03-05T05:34:58Z")
On Tuesday, March 4, 2014 9:32:55 PM UTC-8, siva mannem wrote:
> I installed ES(at the location /usr/lib/elasticsearch/) on our gateway
> server and i am able to run some basic curl commands like XPUT and XGET to
> create some indices and retrieve the data in them.
> i am able to give single line JSON record but i am unable to give JSON
> file as input to curl XPUT .
> can anybody give me the syntax for giving JSON file as input for curl XPUT
> command?
>
> my next issue is i copied the following 4 elasticsearch-hadoop jar files
> elasticsearch-hadoop-1.3.0.M2.jar
> elasticsearch-hadoop-1.3.0.M2-sources.jar
> elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> elasticsearch-hadoop-1.3.0.M2-yarn.jar
>
> to /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
> and /usr/lib/gphd/pig/
>
> i have the following json file j.json
> ++++++
> {"k1":"v1" , "k2":"v2" , "k3":"v3"}
> ++++++++
>
> in my\_hdfs\_path.
>
> my pig script is write\_data\_to\_es.pig
> +++++++++++++
> REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
> DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
> A = LOAD '/my\_hdfs\_path/j.json' using
> JsonLoader('k1:chararray,k2:chararray,k3:chararray');
> STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
> ++++++++++++++
>
> when i run my pig script
> +++++++++
> pig -x mapreduce write\_data\_to\_es.pig
> ++++++++++++
>
> i am getting following error
> +++++++++
> Input(s):
> Failed to read data from "/my\_hdfs\_path/j.json"
>
> Output(s):
> Failed to produce result in "usa/ca"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job\_1390436301987\_0089
>
> 2014-03-05 00:26:50,839 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>
> - Failed!
> 2014-03-05 00:26:50,841 [main] ERROR
> org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
> Failed to read data from "/elastic\_search/es\_hadoop\_test.json"
>
> Output(s):
> Failed to produce result in "usa/ca"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job\_1390436301987\_0089
>
> 2014-03-05 00:26:50,839 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>
> - Failed!
> 2014-03-05 00:26:50,841 [main] ERROR
> org.apache.pig.tools.grunt.GruntParser - _ERROR 2997: Encountered
> IOException. Out of nodes and retries; caught exception_
> Details at logfile:
> /usr/lib/elasticsearch/elasticsearch-0.90.9/pig\_1393997175206.log
> ++++++++++++
>
> i am using pivotal hadoop version (1.0.1) which is basically apache
> hadoop (hadoop-2.0.2)
> and pig version is 0.10.1
> and Elasticsearch version is 0.90.9
>
> can anybody help me out here?
> thank you so much in advance for your help.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7f32cae4-6ba4-4a57-889a-0cd826f69d09%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7f32cae4-6ba4-4a57-889a-0cd826f69d09%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)
#### Post date: [March 5, 2014, 9:39am UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/3 "2014-03-05T09:39:46Z")
Hi,
Is your ES instance known by your Hadoop cluster (/etc/hosts) ?
It does not even seems to read in it.
Cheers,
Yann
Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
> I installed ES(at the location /usr/lib/elasticsearch/) on our gateway
> server and i am able to run some basic curl commands like XPUT and XGET to
> create some indices and retrieve the data in them.
> i am able to give single line JSON record but i am unable to give JSON
> file as input to curl XPUT .
> can anybody give me the syntax for giving JSON file as input for curl XPUT
> command?
>
> my next issue is i copied the following 4 elasticsearch-hadoop jar files
> elasticsearch-hadoop-1.3.0.M2.jar
> elasticsearch-hadoop-1.3.0.M2-sources.jar
> elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> elasticsearch-hadoop-1.3.0.M2-yarn.jar
>
> to /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
> and /usr/lib/gphd/pig/
>
> i have the following json file j.json
> ++++++
> {"k1":"v1" , "k2":"v2" , "k3":"v3"}
> ++++++++
>
> in my\_hdfs\_path.
>
> my pig script is write\_data\_to\_es.pig
> +++++++++++++
> REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
> DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
> A = LOAD '/my\_hdfs\_path/j.json' using
> JsonLoader('k1:chararray,k2:chararray,k3:chararray');
> STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
> ++++++++++++++
>
> when i run my pig script
> +++++++++
> pig -x mapreduce write\_data\_to\_es.pig
> ++++++++++++
>
> i am getting following error
> +++++++++
> Input(s):
> Failed to read data from "/my\_hdfs\_path/j.json"
>
> Output(s):
> Failed to produce result in "usa/ca"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job\_1390436301987\_0089
>
> 2014-03-05 00:26:50,839 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>
> - Failed!
> 2014-03-05 00:26:50,841 [main] ERROR
> org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
> Failed to read data from "/elastic\_search/es\_hadoop\_test.json"
>
> Output(s):
> Failed to produce result in "mannem/siva"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job\_1390436301987\_0089
>
> 2014-03-05 00:26:50,839 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>
> - Failed!
> 2014-03-05 00:26:50,841 [main] ERROR
> org.apache.pig.tools.grunt.GruntParser - _ERROR 2997: Encountered
> IOException. Out of nodes and retries; caught exception_
> Details at logfile:
> /usr/lib/elasticsearch/elasticsearch-0.90.9/pig\_1393997175206.log
> ++++++++++++
>
> i am using pivotal hadoop version (1.0.1) which is basically apache
> hadoop (hadoop-2.0.2)
> and pig version is 0.10.1
> and Elasticsearch version is 0.90.9
>
> can anybody help me out here?
> thank you so much in advance for your help.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3cbde444-460e-4f14-bb16-f160feaf9a58%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3cbde444-460e-4f14-bb16-f160feaf9a58%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [March 5, 2014, 10:50am UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/4 "2014-03-05T10:50:16Z")
The error indicates that Pig cannot access Elasticsearch. Make sure that you specify the proper ES IP and port in your
configuration - the defaults localhost:9200 work only if you are running on a local node or if you have Elasticsearch
running on _each_ node of your Hadoop cluster.
Also when loading Json files, don't use JsonLoader but rather PigStorage - JsonLoader purpose is to transform JSON into
objects but that is not needed.
es-hadoop already does that and if you're data is in json, it will stream the data as is to ES.
Again, I recommend you go through the reference documentation:
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
On 3/5/2014 7:32 AM, siva mannem wrote:
> I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic curl
> commands like XPUT and XGET to create some indices and retrieve the data in them.
> i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
> can anybody give me the syntax for giving JSON file as input for curl XPUT command?
>
> my next issue is i copied the following 4 elasticsearch-hadoop jar files
> elasticsearch-hadoop-1.3.0.M2.jar
> elasticsearch-hadoop-1.3.0.M2-sources.jar
> elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> elasticsearch-hadoop-1.3.0.M2-yarn.jar
>
> to /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
> and /usr/lib/gphd/pig/
>
> i have the following json file j.json
> ++++++
> {"k1":"v1" , "k2":"v2" , "k3":"v3"}
> ++++++++
>
> in my\_hdfs\_path.
>
> my pig script is write\_data\_to\_es.pig
> +++++++++++++
> REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
> DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
> A = LOAD '/my\_hdfs\_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
> STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
> ++++++++++++++
>
> when i run my pig script
> +++++++++
> pig -x mapreduce write\_data\_to\_es.pig
> ++++++++++++
>
> i am getting following error
> +++++++++
> Input(s):
> Failed to read data from "/my\_hdfs\_path/j.json"
>
> Output(s):
> Failed to produce result in "usa/ca"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job\_1390436301987\_0089
>
> 2014-03-05 00:26:50,839 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
> Failed!
> 2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
> Failed to read data from "/elastic\_search/es\_hadoop\_test.json"
>
> Output(s):
> Failed to produce result in "mannem/siva"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job\_1390436301987\_0089
>
> 2014-03-05 00:26:50,839 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
> Failed!
> 2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - _ERROR 2997: Encountered IOException. Out
> of nodes and retries; caught exception_
> Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig\_1393997175206.log
> ++++++++++++
>
> i am using pivotal hadoop version (1.0.1) which is basically apache hadoop (hadoop-2.0.2)
> and pig version is 0.10.1
> and Elasticsearch version is 0.90.9
>
> can anybody help me out here?
> thank you so much in advance for your help.
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/f4740e2a-868f-489c-8f9d-842c08ecddff%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f4740e2a-868f-489c-8f9d-842c08ecddff%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/f4740e2a-868f-489c-8f9d-842c08ecddff%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f4740e2a-868f-489c-8f9d-842c08ecddff%40googlegroups.com?utm_medium=email&utm_source=footer).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
Costin
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53170168.5000909%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/53170168.5000909%40gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@siva\_mannem](https://discuss.elastic.co/u/siva_mannem)
#### Post date: [March 5, 2014, 10:28pm UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/5 "2014-03-05T22:28:50Z")
Yann and Costin,
thank you so much for your quick reply.
now i am able to write data to ES from PIG and also read data from ES
using PIG.
i modified my DEFINE statement in PIG script as follows :
++++++++++
DEFINE ESTOR
org.elasticsearch.hadoop.pig.EsStorage('es.nodes=gateway1,es.resource=usa/ca');
++++++++++++
On Tuesday, March 4, 2014 9:32:55 PM UTC-8, siva mannem wrote:
> I installed ES(at the location /usr/lib/elasticsearch/) on our gateway
> server and i am able to run some basic curl commands like XPUT and XGET to
> create some indices and retrieve the data in them.
> i am able to give single line JSON record but i am unable to give JSON
> file as input to curl XPUT .
> can anybody give me the syntax for giving JSON file as input for curl XPUT
> command?
>
> my next issue is i copied the following 4 elasticsearch-hadoop jar files
> elasticsearch-hadoop-1.3.0.M2.jar
> elasticsearch-hadoop-1.3.0.M2-sources.jar
> elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> elasticsearch-hadoop-1.3.0.M2-yarn.jar
>
> to /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
> and /usr/lib/gphd/pig/
>
> i have the following json file j.json
> ++++++
> {"k1":"v1" , "k2":"v2" , "k3":"v3"}
> ++++++++
>
> in my\_hdfs\_path.
>
> my pig script is write\_data\_to\_es.pig
> +++++++++++++
> REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
> DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
> A = LOAD '/my\_hdfs\_path/j.json' using
> JsonLoader('k1:chararray,k2:chararray,k3:chararray');
> STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
> ++++++++++++++
>
> when i run my pig script
> +++++++++
> pig -x mapreduce write\_data\_to\_es.pig
> ++++++++++++
>
> i am getting following error
> +++++++++
> Input(s):
> Failed to read data from "/my\_hdfs\_path/j.json"
>
> Output(s):
> Failed to produce result in "usa/ca"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job\_1390436301987\_0089
>
> 2014-03-05 00:26:50,839 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>
> - Failed!
> 2014-03-05 00:26:50,841 [main] ERROR
> org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
> Failed to read data from "/elastic\_search/es\_hadoop\_test.json"
>
> Output(s):
> Failed to produce result in "mannem/siva"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job\_1390436301987\_0089
>
> 2014-03-05 00:26:50,839 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>
> - Failed!
> 2014-03-05 00:26:50,841 [main] ERROR
> org.apache.pig.tools.grunt.GruntParser - _ERROR 2997: Encountered
> IOException. Out of nodes and retries; caught exception_
> Details at logfile:
> /usr/lib/elasticsearch/elasticsearch-0.90.9/pig\_1393997175206.log
> ++++++++++++
>
> i am using pivotal hadoop version (1.0.1) which is basically apache
> hadoop (hadoop-2.0.2)
> and pig version is 0.10.1
> and Elasticsearch version is 0.90.9
>
> can anybody help me out here?
> thank you so much in advance for your help.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/18960477-da43-4969-8571-02942ab77390%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/18960477-da43-4969-8571-02942ab77390%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@hanine](https://discuss.elastic.co/u/hanine)
#### Post date: [May 12, 2014, 12:42pm UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/6 "2014-05-12T12:42:31Z")
I had get the same erreur but I don't know what I have to change in my
"/etc/hosts"
thank you for your help
Le mercredi 5 mars 2014 09:39:46 UTC, Yann Barraud a écrit :
> Hi,
>
> Is your ES instance known by your Hadoop cluster (/etc/hosts) ?
>
> It does not even seems to read in it.
>
> Cheers,
> Yann
>
> Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
>
> > I installed ES(at the location /usr/lib/elasticsearch/) on our gateway
> > server and i am able to run some basic curl commands like XPUT and XGET to
> > create some indices and retrieve the data in them.
> > i am able to give single line JSON record but i am unable to give JSON
> > file as input to curl XPUT .
> > can anybody give me the syntax for giving JSON file as input for curl
> > XPUT command?
> >
> > my next issue is i copied the following 4 elasticsearch-hadoop jar files
> > elasticsearch-hadoop-1.3.0.M2.jar
> > elasticsearch-hadoop-1.3.0.M2-sources.jar
> > elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> > elasticsearch-hadoop-1.3.0.M2-yarn.jar
> >
> > to /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
> > and /usr/lib/gphd/pig/
> >
> > i have the following json file j.json
> > ++++++
> > {"k1":"v1" , "k2":"v2" , "k3":"v3"}
> > ++++++++
> >
> > in my\_hdfs\_path.
> >
> > my pig script is write\_data\_to\_es.pig
> > +++++++++++++
> > REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
> > DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
> > A = LOAD '/my\_hdfs\_path/j.json' using
> > JsonLoader('k1:chararray,k2:chararray,k3:chararray');
> > STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
> > ++++++++++++++
> >
> > when i run my pig script
> > +++++++++
> > pig -x mapreduce write\_data\_to\_es.pig
> > ++++++++++++
> >
> > i am getting following error
> > +++++++++
> > Input(s):
> > Failed to read data from "/my\_hdfs\_path/j.json"
> >
> > Output(s):
> > Failed to produce result in "usa/ca"
> >
> > Counters:
> > Total records written : 0
> > Total bytes written : 0
> > Spillable Memory Manager spill count : 0
> > Total bags proactively spilled: 0
> > Total records proactively spilled: 0
> >
> > Job DAG:
> > job\_1390436301987\_0089
> >
> > 2014-03-05 00:26:50,839 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> >
> > - Failed!
> > 2014-03-05 00:26:50,841 [main] ERROR
> > org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
> > Failed to read data from "/elastic\_search/es\_hadoop\_test.json"
> >
> > Output(s):
> > Failed to produce result in "mannem/siva"
> >
> > Counters:
> > Total records written : 0
> > Total bytes written : 0
> > Spillable Memory Manager spill count : 0
> > Total bags proactively spilled: 0
> > Total records proactively spilled: 0
> >
> > Job DAG:
> > job\_1390436301987\_0089
> >
> > 2014-03-05 00:26:50,839 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> >
> > - Failed!
> > 2014-03-05 00:26:50,841 [main] ERROR
> > org.apache.pig.tools.grunt.GruntParser - _ERROR 2997: Encountered
> > IOException. Out of nodes and retries; caught exception_
> > Details at logfile:
> > /usr/lib/elasticsearch/elasticsearch-0.90.9/pig\_1393997175206.log
> > ++++++++++++
> >
> > i am using pivotal hadoop version (1.0.1) which is basically apache
> > hadoop (hadoop-2.0.2)
> > and pig version is 0.10.1
> > and Elasticsearch version is 0.90.9
> >
> > can anybody help me out here?
> > thank you so much in advance for your help.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [May 12, 2014, 3:28pm UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/7 "2014-05-12T15:28:20Z")
Check your network settings and make sure that the Hadoop nodes can communicate with the ES nodes.
If you install ES besides Hadoop itself, this shouldn't be a problem.
There are various way to check this - try ping, tracert, etc...
Please refer to your distro manual/documentation for more information about the configuration and setup.
Cheers,
On 5/12/14 3:42 PM, hanine haninne wrote:
> I had get the same erreur but I don't know what I have to change in my "/etc/hosts"
> thank you for your help
>
> Le mercredi 5 mars 2014 09:39:46 UTC, Yann Barraud a écrit :
>
> ```
> Hi,
>
> Is your ES instance known by your Hadoop cluster (/etc/hosts) ?
>
> It does not even seems to read in it.
>
> Cheers,
> Yann
>
> Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
>
> I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic
> curl commands like XPUT and XGET to create some indices and retrieve the data in them.
> i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
> can anybody give me the syntax for giving JSON file as input for curl XPUT command?
>
> my next issue is i copied the following 4 elasticsearch-hadoop jar files
> elasticsearch-hadoop-1.3.0.M2.jar
> elasticsearch-hadoop-1.3.0.M2-sources.jar
> elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> elasticsearch-hadoop-1.3.0.M2-yarn.jar
>
> to /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
> and /usr/lib/gphd/pig/
>
> i have the following json file j.json
> ++++++
> {"k1":"v1" , "k2":"v2" , "k3":"v3"}
> ++++++++
>
> in my_hdfs_path.
>
> my pig script is write_data_to_es.pig
> +++++++++++++
> REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
> DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
> A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
> STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
> ++++++++++++++
>
> when i run my pig script
> +++++++++
> pig -x mapreduce write_data_to_es.pig
> ++++++++++++
>
> i am getting following error
> +++++++++
> Input(s):
> Failed to read data from "/my_hdfs_path/j.json"
>
> Output(s):
> Failed to produce result in "usa/ca"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job_1390436301987_0089
>
> 2014-03-05 00:26:50,839 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> 2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
> Failed to read data from "/elastic_search/es_hadoop_test.json"
>
> Output(s):
> Failed to produce result in "mannem/siva"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job_1390436301987_0089
>
> 2014-03-05 00:26:50,839 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> 2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - *ERROR 2997: Encountered
> IOException. Out of nodes and retries; caught exception*
> Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
> ++++++++++++
>
> i am using pivotal hadoop version (1.0.1) which is basically apache hadoop (hadoop-2.0.2)
> and pig version is 0.10.1
> and elastic search version is 0.90.9
>
> can anybody help me out here?
> thank you so much in advance for your help.
>
> ```
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm_medium=email&utm_source=footer).
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
Costin
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5370E894.8070308%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5370E894.8070308%40gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@hanine](https://discuss.elastic.co/u/hanine)
#### Post date: [May 12, 2014, 10:20pm UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/8 "2014-05-12T22:20:36Z")
thank you so much for your quick reply,
Here is what I had done
1-instaled hadoop-1.2.1( pig-0.12.0 / hive-0.11.0 /...)
2-download Elasticsearch-1.0.1 and put it in the same file of hadoop
3-copied the following 4 elasticsearch-hadoop jar files
elasticsearch-hadoop-1.3.0.M2.jar
elasticsearch-hadoop-1.3.0.M2-sources.jar
elasticsearch-hadoop-1.3.0.M2-javadoc.jar
elasticsearch-hadoop-1.3.0.M2-yarn.jar
to /pig and hadoop/lib
4- Add them in the PIG\_CLASSPATH
knowing that when I take data from my Desktop and put it in elasticsearch
using pig script it works very well, but when I try to get data from my
HDFS it gives me that :
2014-05-12 23:16:31,765 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR: java.io.IOException:
Out of nodes and retries; caught exception
2014-05-12 23:16:31,765 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2014-05-12 23:16:31,766 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
1.2.1 0.12.0 hduser 2014-05-12 23:15:34 2014-05-12 23:16:31
GROUP\_BY
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job\_201405122310\_0001 weblog\_count,weblog\_group,weblogs
GROUP\_BY,COMBINER Message: Job failed! Error - # of failed Reduce Tasks
exceeded allowed limit. FailedCount: 1. LastFailedTask:
task\_201405122310\_0001\_r\_000000 weblogs1/logs2,
Input(s):
Failed to read data from "/user/weblogs"
Output(s):
Failed to produce result in "weblogs1/logs2"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job\_201405122310\_0001
2014-05-12 23:16:31,766 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
And here is the script :
weblogs = LOAD '/user/weblogs' USING PigStorage('\t')
AS (client\_ip : chararray,
full\_request\_date : chararray,
day : int,
month : chararray,
month\_num : int,
year : int,
hour : int,
minute : int,
second : int,
timezone : chararray,
http\_verb : chararray,
uri : chararray,
http\_status\_code : chararray,
bytes\_returned : chararray,
referrer : chararray,
user\_agent : chararray
);
weblog\_group = GROUP weblogs by (client\_ip, year, month\_num);
weblog\_count = FOREACH weblog\_group GENERATE group.client\_ip, group.year,
group.month\_num, COUNT\_STAR(weblogs) as pageviews;
STORE weblog\_count INTO 'weblogs1/logs2' USING
org.elasticsearch.hadoop.pig.EsStorage();
Le lundi 12 mai 2014 16:28:20 UTC+1, Costin Leau a écrit :
> Check your network settings and make sure that the Hadoop nodes can
> communicate with the ES nodes.
> If you install ES besides Hadoop itself, this shouldn't be a problem.
> There are various way to check this - try ping, tracert, etc...
>
> Please refer to your distro manual/documentation for more information
> about the configuration and setup.
>
> Cheers,
>
> On 5/12/14 3:42 PM, hanine haninne wrote:
>
> > I had get the same erreur but I don't know what I have to change in my
> > "/etc/hosts"
> > thank you for your help
> >
> > Le mercredi 5 mars 2014 09:39:46 UTC, Yann Barraud a écrit :
> >
> > ```
> > Hi,
> >
> > Is your ES instance known by your Hadoop cluster (/etc/hosts) ?
> >
> > It does not even seems to read in it.
> >
> > Cheers,
> > Yann
> >
> > Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
> >
> > I installed ES(at the location /usr/lib/elasticsearch/) on our
> >
> > ```
>
> gateway server and i am able to run some basic
>
> > ```
> > curl commands like XPUT and XGET to create some indices and
> >
> > ```
>
> retrieve the data in them.
>
> > ```
> > i am able to give single line JSON record but i am unable to
> >
> > ```
>
> give JSON file as input to curl XPUT .
>
> > ```
> > can anybody give me the syntax for giving JSON file as input for
> >
> > ```
>
> curl XPUT command?
>
> > ```
> > my next issue is i copied the following 4 elasticsearch-hadoop
> >
> > ```
>
> jar files
>
> > ```
> > elasticsearch-hadoop-1.3.0.M2.jar
> > elasticsearch-hadoop-1.3.0.M2-sources.jar
> > elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> > elasticsearch-hadoop-1.3.0.M2-yarn.jar
> >
> > to /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
> > and /usr/lib/gphd/pig/
> >
> > i have the following json file j.json
> > ++++++
> > {"k1":"v1" , "k2":"v2" , "k3":"v3"}
> > ++++++++
> >
> > in my_hdfs_path.
> >
> > my pig script is write_data_to_es.pig
> > +++++++++++++
> > REGISTER
> >
> > ```
>
> /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
>
> > ```
> > DEFINE ESTOR
> >
> > ```
>
> org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
>
> > ```
> > A = LOAD '/my_hdfs_path/j.json' using
> >
> > ```
>
> JsonLoader('k1:chararray,k2:chararray,k3:chararray');
>
> > ```
> > STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
> > ++++++++++++++
> >
> > when i run my pig script
> > +++++++++
> > pig -x mapreduce write_data_to_es.pig
> > ++++++++++++
> >
> > i am getting following error
> > +++++++++
> > Input(s):
> > Failed to read data from "/my_hdfs_path/j.json"
> >
> > Output(s):
> > Failed to produce result in "usa/ca"
> >
> > Counters:
> > Total records written : 0
> > Total bytes written : 0
> > Spillable Memory Manager spill count : 0
> > Total bags proactively spilled: 0
> > Total records proactively spilled: 0
> >
> > Job DAG:
> > job_1390436301987_0089
> >
> > 2014-03-05 00:26:50,839 [main] INFO
> >
> > ```
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>
> - Failed!
>
> > ```
> > 2014-03-05 00:26:50,841 [main] ERROR
> >
> > ```
>
> org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
>
> > ```
> > Failed to read data from "/elastic_search/es_hadoop_test.json"
> >
> > Output(s):
> > Failed to produce result in "mannem/siva"
> >
> > Counters:
> > Total records written : 0
> > Total bytes written : 0
> > Spillable Memory Manager spill count : 0
> > Total bags proactively spilled: 0
> > Total records proactively spilled: 0
> >
> > Job DAG:
> > job_1390436301987_0089
> >
> > 2014-03-05 00:26:50,839 [main] INFO
> >
> > ```
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>
> - Failed!
>
> > ```
> > 2014-03-05 00:26:50,841 [main] ERROR
> >
> > ```
>
> org.apache.pig.tools.grunt.GruntParser - \*ERROR 2997: Encountered
>
> > ```
> > IOException. Out of nodes and retries; caught exception*
> > Details at logfile:
> >
> > ```
>
> /usr/lib/elasticsearch/elasticsearch-0.90.9/pig\_1393997175206.log
>
> > ```
> > ++++++++++++
> >
> > i am using pivotal hadoop version (1.0.1) which is basically
> >
> > ```
>
> apache hadoop (hadoop-2.0.2)
>
> > ```
> > and pig version is 0.10.1
> > and elastic search version is 0.90.9
> >
> > can anybody help me out here?
> > thank you so much in advance for your help.
> >
> > ```
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ \ > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \\>.
> > To view this discussion on the web visit
>
> [https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com)
>
> > \<
> > [https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com?utm_medium=email&utm_source=footer)\>.
>
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> Costin
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cd9d3143-556a-43c8-9cfd-78b666db48b7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cd9d3143-556a-43c8-9cfd-78b666db48b7%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [May 13, 2014, 9:54am UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/9 "2014-05-13T09:54:17Z")
I would recommend upgrading to the latest es-hadoop, 2.0 RC1.
Also considering reading [1]
Hope this helps,
[1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/troubleshooting.html)
On 5/13/14 1:20 AM, hanine haninne wrote:
> thank you so much for your quick reply,
> Here is what I had done
> 1-instaled hadoop-1.2.1( pig-0.12.0 / hive-0.11.0 /...)
> 2-download Elasticsearch-1.0.1 and put it in the same file of hadoop
> 3-copied the following 4 elasticsearch-hadoop jar files
> elasticsearch-hadoop-1.3.0.M2.jar
> elasticsearch-hadoop-1.3.0.M2-sources.jar
> elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> elasticsearch-hadoop-1.3.0.M2-yarn.jar
> to /pig and hadoop/lib
> 4- Add them in the PIG\_CLASSPATH
>
> knowing that when I take data from my Desktop and put it in elasticsearch using pig script it works very well, but when
> I try to get data from my HDFS it gives me that :
>
> 2014-05-12 23:16:31,765 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR: java.io.IOException: Out of
> nodes and retries; caught exception
> 2014-05-12 23:16:31,765 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2014-05-12 23:16:31,766 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 1.2.1 0.12.0 hduser 2014-05-12 23:15:34 2014-05-12 23:16:31 GROUP\_BY
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> job\_201405122310\_0001 weblog\_count,weblog\_group,weblogs GROUP\_BY,COMBINER Message: Job failed! Error - # of
> failed Reduce Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task\_201405122310\_0001\_r\_000000
> weblogs1/logs2,
>
> Input(s):
> Failed to read data from "/user/weblogs"
>
> Output(s):
> Failed to produce result in "weblogs1/logs2"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job\_201405122310\_0001
>
> 2014-05-12 23:16:31,766 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -
> Failed!
>
> And here is the script :
>
> weblogs = LOAD '/user/weblogs' USING PigStorage('\t')
> AS (client\_ip : chararray,
> full\_request\_date : chararray,
> day : int,
> month : chararray,
> month\_num : int,
> year : int,
> hour : int,
> minute : int,
> second : int,
> timezone : chararray,
> http\_verb : chararray,
> uri : chararray,
> http\_status\_code : chararray,
> bytes\_returned : chararray,
> referrer : chararray,
> user\_agent : chararray
> );
> weblog\_group = GROUP weblogs by (client\_ip, year, month\_num);
> weblog\_count = FOREACH weblog\_group GENERATE group.client\_ip, group.year, group.month\_num, COUNT\_STAR(weblogs) as pageviews;
> STORE weblog\_count INTO 'weblogs1/logs2' USING org.elasticsearch.hadoop.pig.EsStorage();
>
> Le lundi 12 mai 2014 16:28:20 UTC+1, Costin Leau a écrit :
>
> ```
> Check your network settings and make sure that the Hadoop nodes can communicate with the ES nodes.
> If you install ES besides Hadoop itself, this shouldn't be a problem.
> There are various way to check this - try ping, tracert, etc...
>
> Please refer to your distro manual/documentation for more information about the configuration and setup.
>
> Cheers,
>
> On 5/12/14 3:42 PM, hanine haninne wrote:
> > I had get the same erreur but I don't know what I have to change in my "/etc/hosts"
> > thank you for your help
> >
> > Le mercredi 5 mars 2014 09:39:46 UTC, Yann Barraud a écrit :
> >
> > Hi,
> >
> > Is your ES instance known by your Hadoop cluster (/etc/hosts) ?
> >
> > It does not even seems to read in it.
> >
> > Cheers,
> > Yann
> >
> > Le mercredi 5 mars 2014 06:32:55 UTC+1, siva mannem a écrit :
> >
> > I installed ES(at the location /usr/lib/elasticsearch/) on our gateway server and i am able to run some basic
> > curl commands like XPUT and XGET to create some indices and retrieve the data in them.
> > i am able to give single line JSON record but i am unable to give JSON file as input to curl XPUT .
> > can anybody give me the syntax for giving JSON file as input for curl XPUT command?
> >
> > my next issue is i copied the following 4 elasticsearch-hadoop jar files
> > elasticsearch-hadoop-1.3.0.M2.jar
> > elasticsearch-hadoop-1.3.0.M2-sources.jar
> > elasticsearch-hadoop-1.3.0.M2-javadoc.jar
> > elasticsearch-hadoop-1.3.0.M2-yarn.jar
> >
> > to /usr/lib/elasticsearch/elasticsearch-0.90.9/lib
> > and /usr/lib/gphd/pig/
> >
> > i have the following json file j.json
> > ++++++
> > {"k1":"v1" , "k2":"v2" , "k3":"v3"}
> > ++++++++
> >
> > in my_hdfs_path.
> >
> > my pig script is write_data_to_es.pig
> > +++++++++++++
> > REGISTER /usr/lib/gphd/pig/elasticsearch-hadoop-1.3.0.M2-yarn.jar;
> > DEFINE ESTOR org.elasticsearch.hadoop.pig.EsStorage('es.resource=usa/ca');
> > A = LOAD '/my_hdfs_path/j.json' using JsonLoader('k1:chararray,k2:chararray,k3:chararray');
> > STORE A into 'usa/ca' USING ESTOR('es.input.json=true');
> > ++++++++++++++
> >
> > when i run my pig script
> > +++++++++
> > pig -x mapreduce write_data_to_es.pig
> > ++++++++++++
> >
> > i am getting following error
> > +++++++++
> > Input(s):
> > Failed to read data from "/my_hdfs_path/j.json"
> >
> > Output(s):
> > Failed to produce result in "usa/ca"
> >
> > Counters:
> > Total records written : 0
> > Total bytes written : 0
> > Spillable Memory Manager spill count : 0
> > Total bags proactively spilled: 0
> > Total records proactively spilled: 0
> >
> > Job DAG:
> > job_1390436301987_0089
> >
> >
> > 2014-03-05 00:26:50,839 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> > 2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Input(s):
> > Failed to read data from "/elastic_search/es_hadoop_test.json"
> >
> > Output(s):
> > Failed to produce result in "mannem/siva"
> >
> > Counters:
> > Total records written : 0
> > Total bytes written : 0
> > Spillable Memory Manager spill count : 0
> > Total bags proactively spilled: 0
> > Total records proactively spilled: 0
> >
> > Job DAG:
> > job_1390436301987_0089
> >
> > 2014-03-05 00:26:50,839 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> > 2014-03-05 00:26:50,841 [main] ERROR org.apache.pig.tools.grunt.GruntParser - *ERROR 2997: Encountered
> > IOException. Out of nodes and retries; caught exception*
> > Details at logfile: /usr/lib/elasticsearch/elasticsearch-0.90.9/pig_1393997175206.log
> > ++++++++++++
> >
> > i am using pivotal hadoop version (1.0.1) which is basically apache hadoop (hadoop-2.0.2)
> > and pig version is 0.10.1
> > and elastic search version is 0.90.9
> >
> > can anybody help me out here?
> > thank you so much in advance for your help.
> >
> > --
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> >elasticsearc...@googlegroups.com >.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msgid/elasticsearch/1dd8ff7d-ef53-4614-9300-13b5f6ed66fa%40googlegroups.com
>
> > >.
>
> > For more options, visithttps://groups.google.com/d/optout .
>
> --
> Costin
>
> ```
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/cd9d3143-556a-43c8-9cfd-78b666db48b7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cd9d3143-556a-43c8-9cfd-78b666db48b7%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/cd9d3143-556a-43c8-9cfd-78b666db48b7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/cd9d3143-556a-43c8-9cfd-78b666db48b7%40googlegroups.com?utm_medium=email&utm_source=footer).
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
Costin
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5371EBC9.40204%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5371EBC9.40204%40gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:29am UTC](https://discuss.elastic.co/t/unable-to-write-data-to-elasticsearch-using-hadoop-pig/16157/10 "2017-07-06T01:29:52Z")