# Elasticsearch-hadoop: bulk indexing JSON **URL:** \ **Category:** Elasticsearch\ **Created:** [October 6, 2013, 1:19am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-bulk-indexing-json/13858 "2013-10-06T01:19:31Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Douglas\_Moore](https://avatars.discourse-cdn.com/v4/letter/d/46a35a/32.png) [@Douglas\_Moore](https://discuss.elastic.co/u/Douglas_Moore)\ **Post date:** [October 6, 2013, 1:19am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-bulk-indexing-json/13858/1 "2013-10-06T01:19:31Z")
I've prepared files (Gigabytes) of data as JSON and I would like to bulk load these complex JSON documents into ElasticSearch using the elasticsearch-hadoop, using the MapReduce model. Is this possible? The data is organized with a "\_id" field representing the document id and additional fields, some nested, arrays, hashes etc. A simple example: {"\_id":"IX111", "name":"john"}\n {"\_id":"IX112", "name":"jane"}\n {"\_id":"IX113", "name":"jerry"}\n {"\_id":"IX114", "name":"jim"}\n However, in testing It appears that the JSON data is being escaped: 2013-10-05 23:10:37,829 INFO org.elasticsearch.hadoop.rest.BufferedRestClient: Indexing object ["{"\_id":"IN188035438","names":[{"instances":1,"captured\_at":"2013-10-03T01:50:05Z","value":"John"}\t"] which is probably why I see error logs on the Elasticsearch cluster like: 2013-10-05\_21:11:13.78420 [2013-10-05 21:11:13,751][DEBUG][action.bulk] [tv\_es-search-0] [tvperf2v1][20] failed to execute bulk item (index) index {[tvperf2v1][v][yNFpGzvmR2mMruyucT \_e9w], source["{"\_id" ... 2013-10-05\_21:11:13.78410 org.elasticsearch.index.mapper.MapperParsingException: Malformed content, must start with an object 2013-10-05\_21:11:13.78411 at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:479) 2013-10-05\_21:11:13.78412 at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:452) 2013-10-05\_21:11:13.78413 at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:320) 2013-10-05\_21:11:13.78415 at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401) 2013-10-05\_21:11:13.78415 at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155) 2013-10-05\_21:11:13.78416 at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAc tion.java:533) 2013-10-05\_21:11:13.78417 at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:4 18) 2013-10-05\_21:11:13.78418 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 2013-10-05\_21:11:13.78418 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 2013-10-05\_21:11:13.78419 at java.lang.Thread.run(Thread.java:662) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [October 6, 2013, 8:05am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-bulk-indexing-json/13858/2 "2013-10-06T08:05:11Z")
I've replied in the initial issue that you raised. Streaming of JSON documents is not yet supported. As explained in the docs [1], the current M/R model expects the data to be broken down in MapWritable which are then converted into JSON and indexed. In this case, we would have to use a different route - potentially using the Text itself as a the REST payload. Cheers, [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html) On 06/10/2013 4:19 AM, Douglas Moore wrote: > I've prepared files (Gigabytes) of data as JSON and I would like to bulk load these complex JSON documents into > Elasticsearch using the elasticsearch-hadoop, using the MapReduce model. > Is this possible? > > The data is organized with a "\_id" field representing the document id and additional fields, some nested, arrays, hashes > etc. A simple example: > {"\_id":"IX111", "name":"john"}\n > {"\_id":"IX112", "name":"jane"}\n > {"\_id":"IX113", "name":"jerry"}\n > {"\_id":"IX114", "name":"jim"}\n > > However, in testing It appears that the JSON data is being escaped: > > 2013-10-05 23:10:37,829 INFO org.elasticsearch.hadoop.rest.BufferedRestClient: Indexing object ["{"\_id":"IN188035438","names":[{"instances":1,"captured\_at":"2013-10-03T01:50:05Z","value":"John"}\t"] > > which is probably why I see error logs on the Elasticsearch cluster like: > > 2013-10-05\_21:11:13.78420 [2013-10-05 21:11:13,751][DEBUG][action.bulk] [tv\_es-search-0] [tvperf2v1][20] failed to execute bulk item (index) index {[tvperf2v1][v][yNFpGzvmR2mMruyucT > \_e9w], source["{"\_id" ... > > 2013-10-05\_21:11:13.78410 org.elasticsearch.index.mapper.MapperParsingException: Malformed content, must start with an object > 2013-10-05\_21:11:13.78411 at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:479) > 2013-10-05\_21:11:13.78412 at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:452) > 2013-10-05\_21:11:13.78413 at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:320) > 2013-10-05\_21:11:13.78415 at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401) > 2013-10-05\_21:11:13.78415 at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155) > 2013-10-05\_21:11:13.78416 at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAc > tion.java:533) > 2013-10-05\_21:11:13.78417 at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:4 > 18) > 2013-10-05\_21:11:13.78418 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > 2013-10-05\_21:11:13.78418 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > 2013-10-05\_21:11:13.78419 at java.lang.Thread.run(Thread.java:662) > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![James\_Richardson](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@James\_Richardson](https://discuss.elastic.co/u/James_Richardson)\ **Post date:** [October 6, 2013, 8:43am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-bulk-indexing-json/13858/3 "2013-10-06T08:43:30Z")
Why bother with hadoop, if you have less than 1 harddrive worth of data. Just load the files up into es by iterating over the files, with a few threads, you will be done in 15 mins. James -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![M\_20](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m_20/32/1371_2.png) [@M\_20](https://discuss.elastic.co/u/M_20)\ **Post date:** [July 25, 2014, 8:04pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-bulk-indexing-json/13858/4 "2014-07-25T20:04:18Z")
Hi Guys, Could you please give me a java sample code of mapper and reducer in Elasticsearch-hadoop? I'd appreciate it. Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d13f8fa7-fe4c-4070-8c3a-20149cb348eb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d13f8fa7-fe4c-4070-8c3a-20149cb348eb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [July 25, 2014, 8:09pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-bulk-indexing-json/13858/5 "2014-07-25T20:09:22Z")
Have you looked at the docs? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Fri, Jul 25, 2014 at 11:04 PM, M\_20 [rastegar.83@gmail.com](mailto:rastegar.83@gmail.com) wrote: > Hi Guys, > > Could you please give me a java sample code of mapper and reducer in > Elasticsearch-hadoop? > I'd appreciate it. > > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/d13f8fa7-fe4c-4070-8c3a-20149cb348eb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d13f8fa7-fe4c-4070-8c3a-20149cb348eb%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/d13f8fa7-fe4c-4070-8c3a-20149cb348eb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d13f8fa7-fe4c-4070-8c3a-20149cb348eb%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmffdaXvoJHLxEXZLmi6DN%3Deqvs9v7Otisi%2Bqut2V38z2g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmffdaXvoJHLxEXZLmi6DN%3Deqvs9v7Otisi%2Bqut2V38z2g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:13am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-bulk-indexing-json/13858/6 "2017-07-06T01:13:01Z")