# Elasticsearch Hadoop WRITE operation not using Reducer **URL:** \ **Category:** Elasticsearch\ **Created:** [November 1, 2014, 5:20am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-write-operation-not-using-reducer/20525 "2014-11-01T05:20:09Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Sarath](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@Sarath](https://discuss.elastic.co/u/Sarath)\ **Post date:** [November 1, 2014, 5:20am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-write-operation-not-using-reducer/20525/1 "2014-11-01T05:20:09Z")
Hi All, Will Elasticsearch Hadoop WRITE operation doesn't use our custom reducer? I tried with following code and observed that our customer reducer is not invoked. job.setOutputFormatClass(EsOutputFormat.class); job.setMapOutputKeyClass(NullWritable.class); job.setMapOutputValueClass(BytesWritable.class); job.setMapperClass(MyMapper.class); job.setReducerClass(MyReducer.class); ``` configuration.set("es.nodes", "master:9200"); ``` configuration.set("es.resource.write","{indexName}/{indexType}"); configuration.set("es.input.json", "yes"); [//configuration.set](https://configuration.set)("es.write.operation", "upsert"); Thanks, Sarath -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6607380d-48f1-4ba3-9b06-06de0ab0841c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6607380d-48f1-4ba3-9b06-06de0ab0841c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [November 3, 2014, 10:54am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-write-operation-not-using-reducer/20525/2 "2014-11-03T10:54:14Z")
Hi, es-hadoop does not use either mapper or reducers; the Map/Reduce integration relies on the Input/OutputFormat which can be invoked either from a Mapper or from a Reducer. Your Reducer might not be invoked for a variety of reasons; typically the map and reduce phases have different output types and the job fails silently after invoking context.write method. In fact, you can just remove the EsOutputFormat and see whether it makes any difference (it shouldn't). On 11/1/14 7:20 AM, Sarath wrote: > Hi All, > > Will Elasticsearch Hadoop WRITE operation doesn't use our custom reducer? I tried with following code and observed > that our customer reducer is not invoked. > | > job.setOutputFormatClass(EsOutputFormat.class); > job.setMapOutputKeyClass(NullWritable.class); > job.setMapOutputValueClass(BytesWritable.class); > job.setMapperClass(MyMapper.class); > job.setReducerClass(MyReducer.class); > > ``` > configuration.set("es.nodes", "master:9200"); > > ``` > > configuration.set("es.resource.write","{indexName}/{indexType}"); > configuration.set("es.input.json", "yes"); > [//configuration.set](https://configuration.set)("es.write.operation", "upsert"); > | > > Thanks, > Sarath > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/6607380d-48f1-4ba3-9b06-06de0ab0841c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6607380d-48f1-4ba3-9b06-06de0ab0841c%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/6607380d-48f1-4ba3-9b06-06de0ab0841c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6607380d-48f1-4ba3-9b06-06de0ab0841c%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/54575ED6.2010302%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/54575ED6.2010302%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Sarath](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@Sarath](https://discuss.elastic.co/u/Sarath)\ **Post date:** [November 3, 2014, 11:41am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-write-operation-not-using-reducer/20525/3 "2014-11-03T11:41:43Z")
Hi Costin, Thanks for the response. You are right. Map/Reduce integration relies on the Input/OutputFormat. Even after removing EsOutputFormat my custom reducer is not invoked. Should be some issue with hadoop configuration. Thanks, Sarath -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7f0b88d0-fe76-41c2-8c51-398308e913bf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7f0b88d0-fe76-41c2-8c51-398308e913bf%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Telax](https://avatars.discourse-cdn.com/v4/letter/t/d26b3c/32.png) [@Telax](https://discuss.elastic.co/u/Telax)\ **Post date:** [November 4, 2014, 7:46am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-write-operation-not-using-reducer/20525/4 "2014-11-04T07:46:08Z")
Hi, Doesn't look like you've set the number of reduce tasks in your job config. i.e. 'job.setNumReduceTasks(10);' -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1c469ecf-cec7-4d07-840d-d5b91a7a40ae%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1c469ecf-cec7-4d07-840d-d5b91a7a40ae%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Sarath](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@Sarath](https://discuss.elastic.co/u/Sarath)\ **Post date:** [November 4, 2014, 7:50am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-write-operation-not-using-reducer/20525/5 "2014-11-04T07:50:13Z")
Hi Telax, Even though i don't set number of reduce tasks, Hadoop takes care of starting reducers if needed. In my case 1 reducer is running. Issue here custom reducer defined as part of Job configuration is not invoked by Hadoop Thanks, Sarath > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/42616bab-b062-4d61-977a-f845265a94d9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/42616bab-b062-4d61-977a-f845265a94d9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:52am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-write-operation-not-using-reducer/20525/6 "2017-07-06T00:52:22Z")