I end up with the logs being filled with the following error:
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
message contained an invalid tag (zero).
at
com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1398)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1362)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1492)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1487)
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
at
com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
Is it possible that this is caused by the incompatible hadoop versions (2.2
used by plugin with 2.0 being installed) and it is necessary to get it
build with downgraded version?
Also, to build the jar, is it just
gradle build -Pdistro=hadoopYarn ?
Because release 2.0.1 does not quite work for me as it fails to find
JobLocalizer.class.
The hdfs repository relies on vanilla Hadoop 2.2 since that's the official stable version of Yarn. Since you are using a
different
Hadoop version, use the 'light' version as explained in the docs - this contains only the repository-hdfs, without the
Hadoop dependency
(since you already have it installed).
In other words, both the error that you see as well as the 2.0.1 (regarding JobLocalizer) seems to be related to the
differences between
vanilla Hadoop 2.2 and the one you are using.
Hope this helps,
On 8/14/14 7:36 PM, Mateusz Kaczynski wrote:
I'm trying to get es-hadoop repository plugin working on our hadoop 2.0.0-cdh4.6.0 distribution and it seems like I'm
quite lost.
I installed plugin's -hadoop2 version on the machines on our hadoop cluster (which also run our stage elasticsearch nodes).
When attempting to create a repository on one of the datanodes with:
I end up with the logs being filled with the following error:
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1398)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1362)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1492)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1487)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
Is it possible that this is caused by the incompatible hadoop versions (2.2 used by plugin with 2.0 being installed) and
it is necessary to get it build with downgraded version?
Also, to build the jar, is it just
gradle build -Pdistro=hadoopYarn ?
Because release 2.0.1 does not quite work for me as it fails to find JobLocalizer.class.
Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and hadoop/lib
directories in /etc/default/elasticsearch (and exporting it in
/etc/init.d/elasticsearch) and installing light plugin version did work.
On Thursday, 14 August 2014 20:59:39 UTC, Costin Leau wrote:
Hi,
The hdfs repository relies on vanilla Hadoop 2.2 since that's the official
stable version of Yarn. Since you are using a
different
Hadoop version, use the 'light' version as explained in the docs - this
contains only the repository-hdfs, without the
Hadoop dependency
(since you already have it installed).
In other words, both the error that you see as well as the 2.0.1
(regarding JobLocalizer) seems to be related to the
differences between
vanilla Hadoop 2.2 and the one you are using.
Hope this helps,
On 8/14/14 7:36 PM, Mateusz Kaczynski wrote:
I'm trying to get es-hadoop repository plugin working on our hadoop
2.0.0-cdh4.6.0 distribution and it seems like I'm
quite lost.
I installed plugin's -hadoop2 version on the machines on our hadoop
cluster (which also run our stage elasticsearch nodes).
When attempting to create a repository on one of the datanodes with:
I end up with the logs being filled with the following error:
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
message contained an invalid tag (zero).
at
com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
at
com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1398)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.(RpcHeaderProtos.java:1362)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1492)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1487)
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
at
com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at
com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at
org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
Is it possible that this is caused by the incompatible hadoop versions
(2.2 used by plugin with 2.0 being installed) and
it is necessary to get it build with downgraded version?
Also, to build the jar, is it just
gradle build -Pdistro=hadoopYarn ?
Because release 2.0.1 does not quite work for me as it fails to find
JobLocalizer.class.
Regards,
Mateusz
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:> <mailto: elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
To view this discussion on the web visit
Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and hadoop/lib
directories in /etc/default/elasticsearch (and exporting it in
/etc/init.d/elasticsearch) and installing light plugin version did work.
My ES cluster nodes and Hadoop nodes are not collocated. Light version does
not works for me without putting enough correct versions of hadoop related
jars. Right now I don't want to create my jar as Brent did and I don't
want to install hadoop or copy jars on the es nodes either . Right now I
can wait until my hadoop cluster upgrade to a version that es-hadoop
supports. Or there is a version that is compatible to the hadoop My ES
nodes talk to .
Thanks,
On Monday, September 1, 2014 8:50:25 AM UTC-7, Mateusz Kaczynski wrote:
(Much delayed) thank you Costin.
Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and hadoop/lib
directories in /etc/default/elasticsearch (and exporting it in
/etc/init.d/elasticsearch) and installing light plugin version did work.
You need the appropriate hadoop jar on your classpath otherwise
es-hadoop repository plugin cannot connect to HDFS. In the repo,
you'll find two versions with vanilla hadoop1 and hadoop2 - however if
you are using a certain distro, for best compatibility you should use
that distro client jars.
Please refer to your distro client configuration and use the light
version along with those jars.
P.S. In the future, please create your own thread instead of
'hijacking'/reviving an old one.
My ES cluster nodes and Hadoop nodes are not collocated. Light version does
not works for me without putting enough correct versions of hadoop related
jars. Right now I don't want to create my jar as Brent did and I don't want
to install hadoop or copy jars on the es nodes either . Right now I can wait
until my hadoop cluster upgrade to a version that es-hadoop supports. Or
there is a version that is compatible to the hadoop My ES nodes talk to .
Thanks,
On Monday, September 1, 2014 8:50:25 AM UTC-7, Mateusz Kaczynski wrote:
(Much delayed) thank you Costin.
Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and hadoop/lib
directories in /etc/default/elasticsearch (and exporting it in
/etc/init.d/elasticsearch) and installing light plugin version did work.
You need the appropriate hadoop jar on your classpath otherwise
es-hadoop repository plugin cannot connect to HDFS. In the repo,
you'll find two versions with vanilla hadoop1 and hadoop2 - however if
you are using a certain distro, for best compatibility you should use
that distro client jars.
Please refer to your distro client configuration and use the light
version along with those jars.
P.S. In the future, please create your own thread instead of
'hijacking'/reviving an old one.
My ES cluster nodes and Hadoop nodes are not collocated. Light version
does
not works for me without putting enough correct versions of hadoop
related
jars. Right now I don't want to create my jar as Brent did and I don't
want
to install hadoop or copy jars on the es nodes either . Right now I can
wait
until my hadoop cluster upgrade to a version that es-hadoop supports. Or
there is a version that is compatible to the hadoop My ES nodes talk to .
Thanks,
On Monday, September 1, 2014 8:50:25 AM UTC-7, Mateusz Kaczynski wrote:
(Much delayed) thank you Costin.
Indeed, on Ubuntu, changing ES_CLASSPATH to include hadoop and
hadoop/lib
directories in /etc/default/elasticsearch (and exporting it in
/etc/init.d/elasticsearch) and installing light plugin version did work.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.