Questioins about setting up eshadoop

Elastic Stack Elasticsearch
1

Hi all,

I am new to eshadoop and hadoop. I want to contribute to the eshadoop
project but I am having trouble to run the code. I wrote a small piece of
code and followed the step described somewhere online. I add external jar:
elasticsearch-hadoop-1.3.0.BUILD-20130915.230651-143.jar to build path and
export my code to runnable jar in eclipse. But when I run it with: hadoop
jar wordcount, I got class not found exception for esinputformat and
esoutputformat. There are no compiler errors for my code. Here is my code
and the error:
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("es.resource", "twitter/tweet/_search?q=kimchy");
Job job = new Job(conf, "word count");

job.setInputFormatClass(ESInputFormat.class);
job.setOutputFormatClass(ESOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(MapWritable.class);

System.exit(job.waitForCompletion(true) ? 0 : 1);

}

Task Id : attempt_201309170214_0001_m_000006_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.elasticsearch.hadoop.mr.ESOutputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)

Could someone tell me what I did wrong? BTW, I know that a mapreduce
program should have mapper and reducer. But the code(which is similar to
someone's post) above doesn't have. Could someone explain it to me? Thank
you very much!!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

I am getting similar error with :

ES : 1.1.1
Hadoop: Hadoop 1.0.2
JDK: 1.6.0_24
elasticsearch-hadoop-1.3.0.M3

/hadoopApps/wordcount$ hadoop jar WordCount.jar WordCount
/user/himanshu/inputdir
Exception in thread "main" java.lang.NoClassDefFoundError:
org/elasticsearch/hadoop/mr/EsOutputFormat
at WordCount.main(WordCount.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.hadoop.mr.EsOutputFormat
at java.net.UR

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/339c408f-115f-4d27-ba81-079165a420bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

The NCDFE is an indication that the es-hadoop jar is not included in your jar. I recommend reading the official
documentation
[1] which explains each integration in detail, including Map/Reduce.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic

P.S. Make sure you look at the configuration section as well since you are using the old format of "es.resource"

On 5/6/14 8:19 AM, Himanshu Gautam wrote:

I am getting similar error with :

ES : 1.1.1

Hadoop: Hadoop 1.0.2
JDK: 1.6.0_24
elasticsearch-hadoop-1.3.0.M3
/hadoopApps/wordcount$ hadoop jar WordCount.jar WordCount /user/himanshu/inputdir
Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/hadoop/mr/EsOutputFormat
at WordCount.main(WordCount.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.hadoop.mr.EsOutputFormat
at java.net.UR

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/339c408f-115f-4d27-ba81-079165a420bb%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/339c408f-115f-4d27-ba81-079165a420bb%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53688F60.8040601%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

When using the "new" Hadoop API, you need to provide the elasticsearch lib
using the "-libjars" option when running "hadoop jar your.jar
...MainClass". Somehow this first appeared for me after I switched from the
old API to the new API.

Regards,

Jonas

Am Dienstag, 17. September 2013 22:22:58 UTC+2 schrieb Wenfeng Xu:

Hi all,

I am new to eshadoop and hadoop. I want to contribute to the eshadoop
project but I am having trouble to run the code. I wrote a small piece of
code and followed the step described somewhere online. I add external jar:
elasticsearch-hadoop-1.3.0.BUILD-20130915.230651-143.jar to build path
and export my code to runnable jar in eclipse. But when I run it with:
hadoop jar wordcount, I got class not found exception for esinputformat and
esoutputformat. There are no compiler errors for my code. Here is my code
and the error:
public static void main(String args) throws Exception {
Configuration conf = new Configuration();
conf.set("es.resource", "twitter/tweet/_search?q=kimchy");
Job job = new Job(conf, "word count");

job.setInputFormatClass(ESInputFormat.class);
job.setOutputFormatClass(ESOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(MapWritable.class);

System.exit(job.waitForCompletion(true) ? 0 : 1);

}

Task Id : attempt_201309170214_0001_m_000006_0, Status : FAILED

java.lang.RuntimeException: java.lang.ClassNotFoundException: org.elasticsearch.hadoop.mr.ESOutputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)

Could someone tell me what I did wrong? BTW, I know that a mapreduce
program should have mapper and reducer. But the code(which is similar to
someone's post) above doesn't have. Could someone explain it to me? Thank
you very much!!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/23492ffa-818b-433e-9185-ef52e14e8068%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.