# Serialization issue on arrays **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [January 21, 2016, 10:01am UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750 "2016-01-21T10:01:45Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Aurelien\_3](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@Aurelien\_3](https://discuss.elastic.co/u/Aurelien_3)\ **Post date:** [January 21, 2016, 10:01am UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/1 "2016-01-21T10:01:45Z")
Hi all, I'm trying to dump my ES to hadoop to let me work on the data and not bother anymore my cluster. I did a simple job in MR to drop data to HDFS, but it fails with arrays. My ES data has arrays and array might be null or empty sometimes. I get this error : 2016-01-20 18:33:22,679 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException at org.apache.hadoop.io.ArrayWritable.write(ArrayWritable.java:105) at org.elasticsearch.hadoop.mr.WritableArrayWritable.write(WritableArrayWritable.java:60) at org.apache.hadoop.io.MapWritable.write(MapWritable.java:161) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:98) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:82) at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1329) at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat$1.write(SequenceFileOutputFormat.java:83) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) When my mapper is a simple identity mapper. And my configuration : conf.set("es.read.metadata", "false"); conf.set("dfs.replication", "3"); conf.set("mapreduce.client.submit.file.replication", "3"); conf.set("es.nodes.data.only" , "false"); conf.set("es.nodes.discovery", "false"); conf.set("mapreduce.job.maps", "7"); conf.set("es.scroll.keepalive", "20m"); conf.set("es.mapping.include", "id, contributors, contributors\_tags, url"); conf.set("es.field.read.empty.as.null", "true"); do you have any clue on this ? Aurelien ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [January 22, 2016, 8:25am UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/2 "2016-01-22T08:25:11Z")
Looks like there's a bug in Hadoop with `ArrayWritable`s that are empty - what version of ES are you using? ---
**Author:** ![Aurelien\_3](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@Aurelien\_3](https://discuss.elastic.co/u/Aurelien_3)\ **Post date:** [January 22, 2016, 10:13am UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/3 "2016-01-22T10:13:14Z")
I use ES 1.7.2 with the libraries : org.elasticsearch elasticsearch 1.7.2 and org.elasticsearch elasticsearch-hadoop 2.1.2 I'm using Hadoop by HortonWorks : 2.7.1.2.3.2.0-2950 Is it an issue with version compatibility ? ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [January 22, 2016, 10:55am UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/4 "2016-01-22T10:55:42Z")
It's not a compatibility issue, it's a bug in Hadoop (empty ArrayWritables, which are valid and can be constructed cannot be serialized). I've pushed a fix for this in master; the related issue can be found here: > ---
**Author:** ![Aurelien\_3](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@Aurelien\_3](https://discuss.elastic.co/u/Aurelien_3)\ **Post date:** [February 8, 2016, 12:49pm UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/5 "2016-02-08T12:49:10Z")
Actually your fix seems not to work for me. I have this issue : 2016-02-08 12:04:34,101 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException at org.apache.hadoop.io.ArrayWritable.write(ArrayWritable.java:105) at org.elasticsearch.hadoop.mr.WritableArrayWritable.write(WritableArrayWritable.java:60) at org.apache.hadoop.io.MapWritable.write(MapWritable.java:161) at org.apache.hadoop.io.AbstractMapWritable.copy(AbstractMapWritable.java:115) at org.apache.hadoop.io.MapWritable.(MapWritable.java:55) I'm using 2.2.0 version from maven repository. It still get into this part : @Override public void write(DataOutput out) throws IOException { out.writeInt(values.length); // write values for (int i = 0; i \< values.length; i++) { values[i].write(out); } } I don't understand how to handle this. The only way was to clean my ES index prior to do it, but this is not really easy to perform that way. This is not reliable. ---
**Author:** ![Aurelien\_3](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@Aurelien\_3](https://discuss.elastic.co/u/Aurelien_3)\ **Post date:** [February 8, 2016, 8:16pm UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/6 "2016-02-08T20:16:56Z")
Ok, I finally found out the issue. When I have a document with a field array where the array does contains "null" value, I get this error. Actually, this is a new error. First time I got this one. ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [February 21, 2016, 6:26pm UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/7 "2016-02-21T18:26:11Z")
Can you indicate what causes the error in the first place? How do you get a `null` into the array and what is the actual exception? You could raise an issue over at github to track the code fix as well. Thanks, ---
**Author:** ![Aurelien\_3](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@Aurelien\_3](https://discuss.elastic.co/u/Aurelien_3)\ **Post date:** [April 12, 2016, 10:36am UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/8 "2016-04-12T10:36:28Z")
Could not reproduce bug! Thanks for help. ---
**Author:** ![Aurelien\_3](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@Aurelien\_3](https://discuss.elastic.co/u/Aurelien_3)\ **Post date:** [May 23, 2016, 3:39pm UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/9 "2016-05-23T15:39:39Z")
Sorry, I've haven't been able to identify the source of the concrete issue. I've managed to overcome it using input.json : true. Using directly json format reveals to be easier and overcome serialization issue I encountered in Hadoop with ES. Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:24pm UTC](https://discuss.elastic.co/t/serialization-issue-on-arrays/39750/10 "2017-07-06T13:24:39Z")