# Error Writing data to ElasticSearch from PIG

**URL:** <https://discuss.elastic.co/t/error-writing-data-to-elasticsearch-from-pig/17181>\
**Category:** Elasticsearch\
**Created:** [April 24, 2014, 4:35pm UTC](https://discuss.elastic.co/t/error-writing-data-to-elasticsearch-from-pig/17181 "2014-04-24T16:35:28Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ravimbhatt](https://avatars.discourse-cdn.com/v4/letter/r/9de0a6/32.png) [@ravimbhatt](https://discuss.elastic.co/u/ravimbhatt)\
**Post date:** [April 24, 2014, 4:35pm UTC](https://discuss.elastic.co/t/error-writing-data-to-elasticsearch-from-pig/17181/1 "2014-04-24T16:35:28Z")

</div>

Hello All,

I have a very simple pig script.

set default\_parallel 8;

REGISTER 'elasticsearch-hadoop-pig-1.3.0.M3.jar';

sets = LOAD 'wasb://yyyyyy@xxxxxxxxxxxxxxxx' USING PigStorage('\t') AS  
(itemid:chararray, list:chararray);  
a = LIMIT sets 100;  
b = foreach a generate itemid;  
STORE b INTO 'indexOne/iis' USING  
org.elasticsearch.hadoop.pig.EsStorage('[es.nodes=ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com](http://es.nodes=ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com)','es.mapping.names=itemid:@itemid','es.resource=indexOne/iis');

I get below error in my reducer:

java.lang.NullPointerException  
at javax.xml.bind.DatatypeConverterImpl.guessLength(DatatypeConverterImpl.java:653)  
at javax.xml.bind.DatatypeConverterImpl.\_parseBase64Binary(DatatypeConverterImpl.java:691)  
at javax.xml.bind.DatatypeConverterImpl.parseBase64Binary(DatatypeConverterImpl.java:433)  
at javax.xml.bind.DatatypeConverter.parseBase64Binary(DatatypeConverter.java:342)  
at org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:56)  
at org.elasticsearch.hadoop.pig.EsStorage.prepareToWrite(EsStorage.java:178)  
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.(PigOutputFormat.java:125)  
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86)  
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:568)  
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)  
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)  
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:415)  
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)  
at org.apache.hadoop.mapred.Child.main(Child.java:260)

Am i missing something basic here?

Thanks!

Ravi

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [April 24, 2014, 5:06pm UTC](https://discuss.elastic.co/t/error-writing-data-to-elasticsearch-from-pig/17181/2 "2014-04-24T17:06:03Z")

</div>

Hi,

You're not - it looks like a bug inside the base64 function within the JDK (note the input is generated by the same  
class). Out of curiosity what JDK version are you using?

Cheers,

On 4/24/14 7:35 PM, [ravimbhatt@gmail.com](mailto:ravimbhatt@gmail.com) wrote:

> Hello All,
> 
> I have a very simple pig script.
> 
> set default\_parallel 8;
> 
> REGISTER 'elasticsearch-hadoop-pig-1.3.0.M3.jar';
> 
> sets = LOAD 'wasb://yyyyyy@xxxxxxxxxxxxxxxx' USING PigStorage('\t') AS (itemid:chararray, list:chararray);  
> a = LIMIT sets 100;  
> b = foreach a generate itemid;  
> STORE b INTO 'indexOne/iis' USING  
> org.elasticsearch.hadoop.pig.EsStorage('[es.nodes=ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com](http://es.nodes=ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com)','es.mapping.names=itemid:@itemid','es.resource=indexOne/iis');
> 
> I get below error in my reducer:
> 
> java.lang.NullPointerException  
> at javax.xml.bind.DatatypeConverterImpl.guessLength(DatatypeConverterImpl.java:653)  
> at javax.xml.bind.DatatypeConverterImpl.\_parseBase64Binary(DatatypeConverterImpl.java:691)  
> at javax.xml.bind.DatatypeConverterImpl.parseBase64Binary(DatatypeConverterImpl.java:433)  
> at javax.xml.bind.DatatypeConverter.parseBase64Binary(DatatypeConverter.java:342)  
> at org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:56)  
> at org.elasticsearch.hadoop.pig.EsStorage.prepareToWrite(EsStorage.java:178)  
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.(PigOutputFormat.java:125)  
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86)  
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:568)  
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)  
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)  
> at org.apache.hadoop.mapred.Child$4.run(Child.java:266)  
> at java.security.AccessController.doPrivileged(Native Method)  
> at javax.security.auth.Subject.doAs(Subject.java:415)  
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)  
> at org.apache.hadoop.mapred.Child.main(Child.java:260)
> 
> Am i missing something basic here?
> 
> Thanks!
> 
> Ravi
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5359447B.8000708%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5359447B.8000708%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![ravimbhatt](https://avatars.discourse-cdn.com/v4/letter/r/9de0a6/32.png) [@ravimbhatt](https://discuss.elastic.co/u/ravimbhatt)\
**Post date:** [April 24, 2014, 5:09pm UTC](https://discuss.elastic.co/t/error-writing-data-to-elasticsearch-from-pig/17181/3 "2014-04-24T17:09:57Z")

</div>

Thanks for your reply.

My java version is "openjdk version "1.7.0-internal"

Do you suggest trying a different version of java?

On Thursday, 24 April 2014 18:06:03 UTC+1, Costin Leau wrote:

> Hi,
> 
> You're not - it looks like a bug inside the base64 function within the JDK  
> (note the input is generated by the same  
> class). Out of curiosity what JDK version are you using?
> 
> Cheers,
> 
> On 4/24/14 7:35 PM, [ravim...@gmail.com](mailto:ravim...@gmail.com) \<javascript:\> wrote:
> 
> > Hello All,
> > 
> > I have a very simple pig script.
> > 
> > set default\_parallel 8;
> > 
> > REGISTER 'elasticsearch-hadoop-pig-1.3.0.M3.jar';
> > 
> > sets = LOAD 'wasb://yyyyyy@xxxxxxxxxxxxxxxx' USING PigStorage('\t') AS  
> > (itemid:chararray, list:chararray);  
> > a = LIMIT sets 100;  
> > b = foreach a generate itemid;  
> > STORE b INTO 'indexOne/iis' USING  
> > org.elasticsearch.hadoop.pig.EsStorage('es.nodes=  
> > [ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com](http://ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com)','es.mapping.names=itemid:@itemid','es.resource=indexOne/iis');
> 
> > I get below error in my reducer:
> > 
> > java.lang.NullPointerException  
> > at  
> > javax.xml.bind.DatatypeConverterImpl.guessLength(DatatypeConverterImpl.java:653)
> 
> > ```
> > at 
> > 
> > ```
> 
> javax.xml.bind.DatatypeConverterImpl.\_parseBase64Binary(DatatypeConverterImpl.java:691)
> 
> > ```
> > at 
> > 
> > ```
> 
> javax.xml.bind.DatatypeConverterImpl.parseBase64Binary(DatatypeConverterImpl.java:433)
> 
> > ```
> > at 
> > 
> > ```
> 
> javax.xml.bind.DatatypeConverter.parseBase64Binary(DatatypeConverter.java:342)
> 
> > ```
> > at 
> > 
> > ```
> 
> org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:56)
> 
> > ```
> > at 
> > 
> > ```
> 
> org.elasticsearch.hadoop.pig.EsStorage.prepareToWrite(EsStorage.java:178)
> 
> > ```
> > at 
> > 
> > ```
> 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.(PigOutputFormat.java:125)
> 
> > ```
> > at 
> > 
> > ```
> 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86)
> 
> > ```
> > at 
> > 
> > ```
> 
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:568)
> 
> > ```
> > at 
> > 
> > ```
> 
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
> 
> > ```
> > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418) 
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:266) 
> > at java.security.AccessController.doPrivileged(Native Method) 
> > at javax.security.auth.Subject.doAs(Subject.java:415) 
> > at 
> > 
> > ```
> 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)
> 
> > ```
> > at org.apache.hadoop.mapred.Child.main(Child.java:260) 
> > 
> > ```
> > 
> > Am i missing something basic here?
> > 
> > Thanks!
> > 
> > Ravi
> > 
> > --  
> > You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send  
> > an email to  
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\> \<mailto:  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>\>.  
> > To view this discussion on the web visit
> 
> [https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com)
> 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer)\>.
> 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [April 24, 2014, 5:26pm UTC](https://discuss.elastic.co/t/error-writing-data-to-elasticsearch-from-pig/17181/4 "2014-04-24T17:26:13Z")

</div>

I've committed a quick fix and pushed a dev build [1] - I recommend trying that instead which should work on your  
platform as well.

[1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/install.html#download-dev)

P.S. For both Hadoop and ES, Sun/Oracle JDK is recommended.

On 4/24/14 8:09 PM, [ravimbhatt@gmail.com](mailto:ravimbhatt@gmail.com) wrote:

> Thanks for your reply.
> 
> My java version is "openjdk version "1.7.0-internal"
> 
> Do you suggest trying a different version of java?
> 
> On Thursday, 24 April 2014 18:06:03 UTC+1, Costin Leau wrote:
> 
> ```
> Hi,
> 
> You're not - it looks like a bug inside the base64 function within the JDK (note the input is generated by the same
> class). Out of curiosity what JDK version are you using?
> 
> Cheers,
> 
> On 4/24/14 7:35 PM, ravim...@gmail.com <javascript:> wrote:
> > Hello All,
> >
> > I have a very simple pig script.
> >
> > set default_parallel 8;
> >
> > REGISTER 'elasticsearch-hadoop-pig-1.3.0.M3.jar';
> >
> > sets = LOAD 'wasb://yyyyyy@xxxxxxxxxxxxxxxx' USING PigStorage('\t') AS (itemid:chararray, list:chararray);
> > a = LIMIT sets 100;
> > b = foreach a generate itemid;
> > STORE b INTO 'indexOne/iis' USING
> > org.elasticsearch.hadoop.pig.EsStorage('es.nodes=ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com
> <http://ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com>','es.mapping.names=itemid:@itemid','es.resource=indexOne/iis');
> 
> >
> > I get below error in my reducer:
> >
> > java.lang.NullPointerException
> > at javax.xml.bind.DatatypeConverterImpl.guessLength(DatatypeConverterImpl.java:653)
> > at javax.xml.bind.DatatypeConverterImpl._parseBase64Binary(DatatypeConverterImpl.java:691)
> > at javax.xml.bind.DatatypeConverterImpl.parseBase64Binary(DatatypeConverterImpl.java:433)
> > at javax.xml.bind.DatatypeConverter.parseBase64Binary(DatatypeConverter.java:342)
> > at org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:56)
> > at org.elasticsearch.hadoop.pig.EsStorage.prepareToWrite(EsStorage.java:178)
> > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.<init>(PigOutputFormat.java:125)
> > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86)
> > at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:568)
> > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
> > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)
> > at org.apache.hadoop.mapred.Child.main(Child.java:260)
> >
> >
> > Am i missing something basic here?
> >
> >
> > Thanks!
> >
> > Ravi
> >
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> >elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com>
> > <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> 
> > For more options, visithttps://groups.google.com/d/optout <https://groups.google.com/d/optout>.
> 
> --
> Costin
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53594935.2070002%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/53594935.2070002%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![ravimbhatt](https://avatars.discourse-cdn.com/v4/letter/r/9de0a6/32.png) [@ravimbhatt](https://discuss.elastic.co/u/ravimbhatt)\
**Post date:** [April 24, 2014, 9:22pm UTC](https://discuss.elastic.co/t/error-writing-data-to-elasticsearch-from-pig/17181/5 "2014-04-24T21:22:51Z")

</div>

Hi Costin,

Thanks for the fix. I tried that and now i get a new error:

java.lang.NullPointerException  
at org.codehaus.jackson.node.TextNode.getBinaryValue(TextNode.java:69)  
at org.codehaus.jackson.node.TextNode.getBinaryValue(TextNode.java:160)  
at org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:71)  
at org.elasticsearch.hadoop.pig.EsStorage.prepareToWrite(EsStorage.java:178)  
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.(PigOutputFormat.java:125)  
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86)  
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:568)  
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)  
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)  
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:415)  
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)  
at org.apache.hadoop.mapred.Child.main(Child.java:260)

may be missing some more null checks in the code? Can you please help?

Thanks!

Ravi

On Thursday, 24 April 2014 18:26:13 UTC+1, Costin Leau wrote:

> I've committed a quick fix and pushed a dev build [1] - I recommend trying  
> that instead which should work on your  
> platform as well.
> 
> [1]  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/install.html#download-dev)
> 
> P.S. For both Hadoop and ES, Sun/Oracle JDK is recommended.
> 
> On 4/24/14 8:09 PM, [ravim...@gmail.com](mailto:ravim...@gmail.com) \<javascript:\> wrote:
> 
> > Thanks for your reply.
> > 
> > My java version is "openjdk version "1.7.0-internal"
> > 
> > Do you suggest trying a different version of java?
> > 
> > On Thursday, 24 April 2014 18:06:03 UTC+1, Costin Leau wrote:
> > 
> > ```
> > Hi, 
> > 
> > You're not - it looks like a bug inside the base64 function within 
> > 
> > ```
> 
> the JDK (note the input is generated by the same
> 
> > ```
> > class). Out of curiosity what JDK version are you using? 
> > 
> > Cheers, 
> > 
> > On 4/24/14 7:35 PM, ravim...@gmail.com <javascript:> wrote: 
> > > Hello All, 
> > > 
> > > I have a very simple pig script. 
> > > 
> > > set default_parallel 8; 
> > > 
> > > REGISTER 'elasticsearch-hadoop-pig-1.3.0.M3.jar'; 
> > > 
> > > sets = LOAD 'wasb://yyyyyy@xxxxxxxxxxxxxxxx' USING 
> > 
> > ```
> 
> PigStorage('\t') AS (itemid:chararray, list:chararray);
> 
> > ```
> > > a = LIMIT sets 100; 
> > > b = foreach a generate itemid; 
> > > STORE b INTO 'indexOne/iis' USING 
> > > org.elasticsearch.hadoop.pig.EsStorage('es.nodes=
> > 
> > ```
> 
> [ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com](http://ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com)
> 
> > ```
> > <http://ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com>','es.mapping.names=itemid:@itemid','es.resource=indexOne/iis'); 
> > 
> > ```
> 
> > ```
> > > 
> > > I get below error in my reducer: 
> > > 
> > > java.lang.NullPointerException 
> > > at 
> > 
> > ```
> 
> javax.xml.bind.DatatypeConverterImpl.guessLength(DatatypeConverterImpl.java:653)
> 
> > ```
> > > at 
> > 
> > ```
> 
> javax.xml.bind.DatatypeConverterImpl.\_parseBase64Binary(DatatypeConverterImpl.java:691)
> 
> > ```
> > > at 
> > 
> > ```
> 
> javax.xml.bind.DatatypeConverterImpl.parseBase64Binary(DatatypeConverterImpl.java:433)
> 
> > ```
> > > at 
> > 
> > ```
> 
> javax.xml.bind.DatatypeConverter.parseBase64Binary(DatatypeConverter.java:342)
> 
> > ```
> > > at 
> > 
> > ```
> 
> org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:56)
> 
> > ```
> > > at 
> > 
> > ```
> 
> org.elasticsearch.hadoop.pig.EsStorage.prepareToWrite(EsStorage.java:178)
> 
> > ```
> > > at 
> > 
> > ```
> 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.(PigOutputFormat.java:125)
> 
> > ```
> > > at 
> > 
> > ```
> 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86)
> 
> > ```
> > > at 
> > 
> > ```
> 
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:568)
> 
> > ```
> > > at 
> > 
> > ```
> 
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
> 
> > ```
> > > at 
> > 
> > ```
> 
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> 
> > ```
> > > at org.apache.hadoop.mapred.Child$4.run(Child.java:266) 
> > > at java.security.AccessController.doPrivileged(Native 
> > 
> > ```
> 
> Method)
> 
> > ```
> > > at javax.security.auth.Subject.doAs(Subject.java:415) 
> > > at 
> > 
> > ```
> 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)
> 
> > ```
> > > at org.apache.hadoop.mapred.Child.main(Child.java:260) 
> > > 
> > > 
> > > Am i missing something basic here? 
> > > 
> > > 
> > > Thanks! 
> > > 
> > > Ravi 
> > > 
> > > 
> > > 
> > > 
> > > -- 
> > > You received this message because you are subscribed to the Google 
> > 
> > ```
> 
> Groups "elasticsearch" group.
> 
> > ```
> > > To unsubscribe from this group and stop receiving emails from it, 
> > 
> > ```
> 
> send an email to
> 
> > ```
> > >elasticsearc...@googlegroups.com <javascript:> <mailto:
> > 
> > ```
> 
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\> \<javascript:\>\>.
> 
> > ```
> > > To view this discussion on the web visit 
> > >
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com)
> 
> > ```
> > <
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com)\>
> 
> > ```
> > > <
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer)
> 
> > ```
> > <
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer)\>\>.
> 
> > ```
> > > For more options, visithttps://groups.google.com/d/optout <
> > 
> > ```
> 
> [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>.
> 
> > ```
> > -- 
> > Costin 
> > 
> > ```
> > 
> > --  
> > You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send  
> > an email to  
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\> \<mailto:  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<javascript:\>\>.  
> > To view this discussion on the web visit
> 
> [https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com)
> 
> > \<  
> > [https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com?utm_medium=email&utm_source=footer)\>.
> 
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/be613c65-9c01-4190-9b0a-f0d18500c39d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be613c65-9c01-4190-9b0a-f0d18500c39d%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [April 25, 2014, 1:15pm UTC](https://discuss.elastic.co/t/error-writing-data-to-elasticsearch-from-pig/17181/6 "2014-04-25T13:15:19Z")

</div>

Hi,

I've pushed another fix however it's probably just a band-aid. For some reason your Pig engine does not set the schema  
(from b) on the OutputWriter.  
I've tried reproducing the error but I couldn't - it would be great if you could join the IRC 1 to track  
down the source of the problem.  
By the way, what is the Pig version that you are using?

Cheers,

[1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/community/)

On 4/25/14 12:22 AM, [ravimbhatt@gmail.com](mailto:ravimbhatt@gmail.com) wrote:

> Hi Costin,
> 
> Thanks for the fix. I tried that and now i get a new error:
> 
> java.lang.NullPointerException  
> at org.codehaus.jackson.node.TextNode.getBinaryValue(TextNode.java:69)  
> at org.codehaus.jackson.node.TextNode.getBinaryValue(TextNode.java:160)  
> at org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:71)  
> at org.elasticsearch.hadoop.pig.EsStorage.prepareToWrite(EsStorage.java:178)  
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.(PigOutputFormat.java:125)  
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86)  
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:568)  
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)  
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)  
> at org.apache.hadoop.mapred.Child$4.run(Child.java:266)  
> at java.security.AccessController.doPrivileged(Native Method)  
> at javax.security.auth.Subject.doAs(Subject.java:415)  
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)  
> at org.apache.hadoop.mapred.Child.main(Child.java:260)
> 
> may be missing some more null checks in the code? Can you please help?
> 
> Thanks!
> 
> Ravi
> 
> On Thursday, 24 April 2014 18:26:13 UTC+1, Costin Leau wrote:
> 
> ```
> I've committed a quick fix and pushed a dev build [1] - I recommend trying that instead which should work on your
> platform as well.
> 
> [1] http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/install.html#download-dev
> <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/install.html#download-dev>
> 
> P.S. For both Hadoop and ES, Sun/Oracle JDK is recommended.
> 
> On 4/24/14 8:09 PM, ravim...@gmail.com <javascript:> wrote:
> > Thanks for your reply.
> >
> > My java version is "openjdk version "1.7.0-internal"
> >
> > Do you suggest trying a different version of java?
> >
> > On Thursday, 24 April 2014 18:06:03 UTC+1, Costin Leau wrote:
> >
> > Hi,
> >
> > You're not - it looks like a bug inside the base64 function within the JDK (note the input is generated by the same
> > class). Out of curiosity what JDK version are you using?
> >
> > Cheers,
> >
> > On 4/24/14 7:35 PM,ravim...@gmail.com <javascript:> wrote:
> > > Hello All,
> > >
> > > I have a very simple pig script.
> > >
> > > set default_parallel 8;
> > >
> > > REGISTER 'elasticsearch-hadoop-pig-1.3.0.M3.jar';
> > >
> > > sets = LOAD 'wasb://yyyyyy@xxxxxxxxxxxxxxxx' USING PigStorage('\t') AS (itemid:chararray, list:chararray);
> > > a = LIMIT sets 100;
> > > b = foreach a generate itemid;
> > > STORE b INTO 'indexOne/iis' USING
> > > org.elasticsearch.hadoop.pig.EsStorage('es.nodes=ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com
> <http://ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com>
> > <http://ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com
> <http://ec2-xx-xx-xxx-xxx.eu-west-1.compute.amazonaws.com>>','es.mapping.names=itemid:@itemid','es.resource=indexOne/iis');
> 
> >
> > >
> > > I get below error in my reducer:
> > >
> > > java.lang.NullPointerException
> > > at javax.xml.bind.DatatypeConverterImpl.guessLength(DatatypeConverterImpl.java:653)
> > > at javax.xml.bind.DatatypeConverterImpl._parseBase64Binary(DatatypeConverterImpl.java:691)
> > > at javax.xml.bind.DatatypeConverterImpl.parseBase64Binary(DatatypeConverterImpl.java:433)
> > > at javax.xml.bind.DatatypeConverter.parseBase64Binary(DatatypeConverter.java:342)
> > > at org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:56)
> > > at org.elasticsearch.hadoop.pig.EsStorage.prepareToWrite(EsStorage.java:178)
> > > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.<init>(PigOutputFormat.java:125)
> > > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:86)
> > > at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:568)
> > > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
> > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> > > at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:415)
> > > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:260)
> > >
> > >
> > > Am i missing something basic here?
> > >
> > >
> > > Thanks!
> > >
> > > Ravi
> > >
> > >
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to
> > >elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:>
> <javascript:>>.
> > > To view this discussion on the web visit
> > >https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com>
> > <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com>>
> > > <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer>
> 
> > <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/e6c6ba6d-012b-49b0-b6a7-785e001d34b9%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
> 
> >
> > > For more options, visithttps://groups.google.com/d/optout <http://groups.google.com/d/optout> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > Costin
> >
> > --
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> >elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com>
> > <https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/elasticsearch/1df51b7e-2cb2-44d9-8a9b-dfb493a352e2%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> 
> > For more options, visithttps://groups.google.com/d/optout <https://groups.google.com/d/optout>.
> 
> --
> Costin
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/be613c65-9c01-4190-9b0a-f0d18500c39d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be613c65-9c01-4190-9b0a-f0d18500c39d%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/be613c65-9c01-4190-9b0a-f0d18500c39d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/be613c65-9c01-4190-9b0a-f0d18500c39d%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/535A5FE7.6010501%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/535A5FE7.6010501%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:33am UTC](https://discuss.elastic.co/t/error-writing-data-to-elasticsearch-from-pig/17181/7 "2017-07-06T01:33:32Z")

</div>