Elasticsearch-hadoop and Apache Storm integration

Elastic Stack Elasticsearch
1

Hello there,

I am following https://www.elastic.co/guide/en/elasticsearch/hadoop/current/storm.html .

I am trying to write to elastic search using the ESBolt. The flow's topology consists of only 1 spout and 1 bolt
builder.setSpout("spout", new RandomSentenceSpout(), 10);
builder.setBolt("es-bolt", new EsBolt("storm/docs"), 5) .shuffleGrouping("spout");

with conf.put("es.index.auto.create", "true");

However, when i run the flow, no errors, no issues. I can see that it successfully created the storm/docs in es. However, it does not index anything.

Tried debugging through the code. Inside the TemplatedBulk.class, on one of the indexing request, i see that the write method is returning:

{"index":{}}
{"word":"snow white and the seven dwarfs"}

Should the "index" entry be empty? Any suggestions on how to figure out what I am doing wrong?

Env info:
hadoop: 2.4
storm: .10
ES: 2.3.3

thanks
Adisa

2

hello there. anybody? @costin , @james.baiera any suggestions?

3

Hey there!

To give some wisdom on the bulk output - bulk data doesn't really need the index/type specified on it in this case because that data is being provided as part of the resource path on the REST call that the connector makes.

As for why documents are not being indexed, I don't think there's enough information to go off of in the post. I would suggest double checking your mappings and index settings as a reasonable starting point.

Hope that helps!

4

FYI even the devs that answer on here do so on their own time, so please be patient :slight_smile: