# Trouble With Elasticsearch Hadoop Integration **URL:** \ **Category:** Elasticsearch\ **Created:** [September 13, 2011, 9:40pm UTC](https://discuss.elastic.co/t/trouble-with-elasticsearch-hadoop-integration/5375 "2011-09-13T21:40:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Ed\_Brown\_VSTI](https://avatars.discourse-cdn.com/v4/letter/e/77aa72/32.png) [@Ed\_Brown\_VSTI](https://discuss.elastic.co/u/Ed_Brown_VSTI)\ **Post date:** [September 13, 2011, 9:40pm UTC](https://discuss.elastic.co/t/trouble-with-elasticsearch-hadoop-integration/5375/1 "2011-09-13T21:40:28Z")
Hello, I'm having trouble with the integration of Elasticsearch and Cloudera's version of Hadoop. Elasticsearch fails to start. Elasticsearch version: 0.17.6 Cloudera Hadoop: 0.20.1 I also installed the Hadoop plugin by using the following: plugin -install hadoop The following lines are in the elasticsearch.yaml file: === gateway.type: hdfs gateway.hdfs.uri: hdfs://127.0.0.1:8020 gateway.hdfs.path: /user/hdfs/output\_multi === I have verified the Hadoop name node installation by creating files in HDFS and using the name node to list the files. The output is as follows: ==== Hadoop command output start ==== hadoop fs -fs hdfs://127.0.0.1:8020 -ls /user/hdfs/output\_multi Found 3 items -rw-r--r-- 1 hdfs supergroup 0 2011-09-13 17:17 /user/hdfs/ output\_multi/\_SUCCESS drwxr-xr-x - hdfs supergroup 0 2011-09-13 17:05 /user/hdfs/ output\_multi/\_logs -rw-r--r-- 1 hdfs supergroup 129962 2011-09-13 17:14 /user/hdfs/ output\_multi/part-r-00000 ==== Hadoop command output end ==== When I start Elasticsearch, it fails with the following error. == Error in Elasticsearch log == {elasticsearch/0.17.6}: Initialization Failed ... 1. IOException[Call to localhost/127.0.0.1:8020 failed on local exception: java.io.EOFException] == End Elasticsearch log error == At the same time, the following error appears in the Hadoop name node log file: == Error in Hadoop name node log == 2011-09-13 17:20:34,439 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 127.0.0.1:35627 got version 3 expected version 4 2011-09-13 17:20:34,566 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 127.0.0.1:35628 got version 3 expected version 4 == End Hadoop name node log == Any help would be appreciated. TYIA ---
**Author:** ![Ed\_Brown\_VSTI](https://avatars.discourse-cdn.com/v4/letter/e/77aa72/32.png) [@Ed\_Brown\_VSTI](https://discuss.elastic.co/u/Ed_Brown_VSTI)\ **Post date:** [September 13, 2011, 9:44pm UTC](https://discuss.elastic.co/t/trouble-with-elasticsearch-hadoop-integration/5375/2 "2011-09-13T21:44:22Z")
Sorry, it's Coludera Hadoop: 0.20.2 On Sep 13, 5:40 pm, Ed Brown VSTI [edbrownv...@gmail.com](mailto:edbrownv...@gmail.com) wrote: > Hello, > I'm having trouble with the integration of Elasticsearch and > Cloudera's version of Hadoop. Elasticsearch fails to start. > > Elasticsearch version: 0.17.6 > Cloudera Hadoop: 0.20.1 > > I also installed the Hadoop plugin by using the following: > plugin -install hadoop > > The following lines are in the elasticsearch.yaml file: > > === > > gateway.type: hdfs > gateway.hdfs.uri: hdfs://127.0.0.1:8020 > gateway.hdfs.path: /user/hdfs/output\_multi > > === > > I have verified the Hadoop name node installation by creating files in > HDFS and using the name node to list the files. The output is as > follows: > > ==== Hadoop command output start ==== > > hadoop fs -fs hdfs://127.0.0.1:8020 -ls /user/hdfs/output\_multi > Found 3 items > -rw-r--r-- 1 hdfs supergroup 0 2011-09-13 17:17 /user/hdfs/ > output\_multi/\_SUCCESS > drwxr-xr-x - hdfs supergroup 0 2011-09-13 17:05 /user/hdfs/ > output\_multi/\_logs > -rw-r--r-- 1 hdfs supergroup 129962 2011-09-13 17:14 /user/hdfs/ > output\_multi/part-r-00000 > > ==== Hadoop command output end ==== > > When I start Elasticsearch, it fails with the following error. > > == Error in Elasticsearch log == > > {elasticsearch/0.17.6}: Initialization Failed ... > > 1. IOException[Call to localhost/127.0.0.1:8020 failed on local > exception: java.io.EOFException] > > == End Elasticsearch log error == > > At the same time, the following error appears in the Hadoop name node > log file: > > == Error in Hadoop name node log == > > 2011-09-13 17:20:34,439 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 127.0.0.1:35627 got version 3 expected > version 4 > 2011-09-13 17:20:34,566 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 127.0.0.1:35628 got version 3 expected > version 4 > > == End Hadoop name node log == > > Any help would be appreciated. > > TYIA ---
**Author:** ![philwhln](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philwhln/32/3121_2.png) [@philwhln](https://discuss.elastic.co/u/philwhln)\ **Post date:** [September 14, 2011, 5:00am UTC](https://discuss.elastic.co/t/trouble-with-elasticsearch-hadoop-integration/5375/3 "2011-09-14T05:00:24Z")
Hi Ed, On Tue, Sep 13, 2011 at 2:40 PM, Ed Brown VSTI [edbrownvsti@gmail.com](mailto:edbrownvsti@gmail.com)wrote: > Elasticsearch version: 0.17.6 > Cloudera Hadoop: 0.20.1 > > == Error in Hadoop name node log == > > 2011-09-13 17:20:34,439 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 127.0.0.1:35627 got version 3 expected > version 4 Sounds like you have mismatching versions of Hadoop jars. Elasticsearch 0.17.6 uses hadoop-core:0.20.2, whereas you're using Apache Hadoop 0.20.1 (via Cloudera's CDH2) Install Cloudera's CDH3 and you should be all good. Cheers, Phil [http://www.philwhln.com](http://www.philwhln.com) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 14, 2011, 7:48am UTC](https://discuss.elastic.co/t/trouble-with-elasticsearch-hadoop-integration/5375/4 "2011-09-14T07:48:49Z")
Another option is to copy the hadoop jar into the plugin instead of the one elasticsearch comes with. On Wed, Sep 14, 2011 at 8:00 AM, Phil Whelan [phil123@gmail.com](mailto:phil123@gmail.com) wrote: > Hi Ed, > > On Tue, Sep 13, 2011 at 2:40 PM, Ed Brown VSTI [edbrownvsti@gmail.com](mailto:edbrownvsti@gmail.com)wrote: > > > Elasticsearch version: 0.17.6 > > Cloudera Hadoop: 0.20.1 > > > > == Error in Hadoop name node log == > > > > 2011-09-13 17:20:34,439 WARN org.apache.hadoop.ipc.Server: Incorrect > > header or version mismatch from 127.0.0.1:35627 got version 3 expected > > version 4 > > Sounds like you have mismatching versions of Hadoop jars. > > Elasticsearch 0.17.6 uses hadoop-core:0.20.2, whereas you're using Apache > Hadoop 0.20.1 (via Cloudera's CDH2) > > Install Cloudera's CDH3 and you should be all good. > > Cheers, > Phil > > [http://www.philwhln.com](http://www.philwhln.com) ---
**Author:** ![Ed\_Brown\_VSTI](https://avatars.discourse-cdn.com/v4/letter/e/77aa72/32.png) [@Ed\_Brown\_VSTI](https://discuss.elastic.co/u/Ed_Brown_VSTI)\ **Post date:** [September 14, 2011, 1:07pm UTC](https://discuss.elastic.co/t/trouble-with-elasticsearch-hadoop-integration/5375/5 "2011-09-14T13:07:20Z")
Phil, That was EXACTLY the problem! The Hadoop jar that was in the plugin directory was MUCH SMALLER than the Hadoop jar provided by Cloudera. Once I switched jars, all was well! On Sep 14, 1:00 am, Phil Whelan [phil...@gmail.com](mailto:phil...@gmail.com) wrote: > Hi Ed, > > On Tue, Sep 13, 2011 at 2:40 PM, Ed Brown VSTI [edbrownv...@gmail.com](mailto:edbrownv...@gmail.com)wrote: > > > Elasticsearch version: 0.17.6 > > Cloudera Hadoop: 0.20.1 > > > == Error in Hadoop name node log == > > > 2011-09-13 17:20:34,439 WARN org.apache.hadoop.ipc.Server: Incorrect > > header or version mismatch from 127.0.0.1:35627 got version 3 expected > > version 4 > > Sounds like you have mismatching versions of Hadoop jars. > > Elasticsearch 0.17.6 uses hadoop-core:0.20.2, whereas you're using Apache > Hadoop 0.20.1 (via Cloudera's CDH2) > > Install Cloudera's CDH3 and you should be all good. > > Cheers, > Phil > > [http://www.philwhln.com](http://www.philwhln.com) ---
**Author:** ![Ed\_Brown\_VSTI](https://avatars.discourse-cdn.com/v4/letter/e/77aa72/32.png) [@Ed\_Brown\_VSTI](https://discuss.elastic.co/u/Ed_Brown_VSTI)\ **Post date:** [September 14, 2011, 1:08pm UTC](https://discuss.elastic.co/t/trouble-with-elasticsearch-hadoop-integration/5375/6 "2011-09-14T13:08:06Z")
Yes, I switched the plugin jar and all was well. I march on! 😃 On Sep 14, 3:48 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Another option is to copy the hadoop jar into the plugin instead of the one > elasticsearch comes with. > > On Wed, Sep 14, 2011 at 8:00 AM, Phil Whelan [phil...@gmail.com](mailto:phil...@gmail.com) wrote: > > > Hi Ed, > > > On Tue, Sep 13, 2011 at 2:40 PM, Ed Brown VSTI [edbrownv...@gmail.com](mailto:edbrownv...@gmail.com)wrote: > > > > Elasticsearch version: 0.17.6 > > > Cloudera Hadoop: 0.20.1 > > > > == Error in Hadoop name node log == > > > > 2011-09-13 17:20:34,439 WARN org.apache.hadoop.ipc.Server: Incorrect > > > header or version mismatch from 127.0.0.1:35627 got version 3 expected > > > version 4 > > > Sounds like you have mismatching versions of Hadoop jars. > > > Elasticsearch 0.17.6 uses hadoop-core:0.20.2, whereas you're using Apache > > Hadoop 0.20.1 (via Cloudera's CDH2) > > > Install Cloudera's CDH3 and you should be all good. > > > Cheers, > > Phil > > > [http://www.philwhln.com](http://www.philwhln.com) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:54am UTC](https://discuss.elastic.co/t/trouble-with-elasticsearch-hadoop-integration/5375/7 "2017-07-06T03:54:24Z")