ES 2.0.0 + ES-Hadoop 2.1.1 | 2.2.0-m1 usecase & issues


0/ HEAD vs HQ
Always the same problem with HEAD plugin (connect button -> error "_status"), anyway I use now HQ and everything is find.

1/ node.mater, vs
In previous release (1.7.X) I use the properties node.master and to tune every nodes.
In the 2.0.0. there's only a property. HQ show the nodes (Screenshot 04) :
This configuration is very important to tune multi node Architecture.
How data node are configured in yml file ?

2/ hadoop + pig + elastic
I use elasticsearch-hadoop.
Yesterday Architecture : pig-0.15.0, ES 1.7.2 (2 nodes), elasticsearch-hadoop-2.1.1, Kibana 4.1.2.

REGISTER /jbigdata/elasticsearch-hadoop-2.1.1/dist/elasticsearch-hadoop-pig-2.1.1.jar;
-- REGISTER /jbigdata/elasticsearch-hadoop-2.2.0-m1/dist/elasticsearch-hadoop-pig-2.2.0-m1.jar;

Everything works fine : Pig latin script, ES index creation, Kibana dashboards.

Nowdays Architecture : pig-0.15.0, ES 2.0.0 (2 nodes), elasticsearch-hadoop-2.1.1 (elasticsearch-hadoop-pig-2.2.0-m1, same issue), Kibana 4.X (not yet used).

When I use the org.elasticsearch.hadoop.pig.EsStorage(...) (works fine with ES 1.7.2), it fails:
The hadoop JobHistory (Screenshot 03) shows the issue:
2 jobs OK with ES 1.7.2, the same PIG script, 2 jobs KO with ES 2.0.0, stacktrace:

2015-10-29 09:20:15,296 INFO [IPC Server handler 0 on 48708] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1445412791627_0003_m_000003 given task: attempt_1445412791627_0003_m_000000_0
2015-10-29 09:20:18,757 INFO [IPC Server handler 1 on 48708] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1445412791627_0003_m_000000_0 is : 0.0
2015-10-29 09:20:18,778 FATAL [IPC Server handler 12 on 48708] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1445412791627_0003_m_000000_0 - exited : java.lang.StringIndexOutOfBoundsException: String index out of range: -29
	at java.lang.String.substring(
	at org.elasticsearch.hadoop.pig.EsStorage.putNext(
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(
	at org.apache.hadoop.mapred.MapTask.runNewMapper(
	at org.apache.hadoop.mapred.YarnChild$
	at Method)
	at org.apache.hadoop.mapred.YarnChild.main(

Any Idea ?

It's probably easier if you create a new thread for your Hadoop issues in the Hadoop forum.

Not sure why you think this - Node | Elasticsearch Guide [8.11] | Elastic

OK, I keep on investiguate.
This OpenSource software is cool.