Mapping es.mapping.id

Elastic Stack Elasticsearch
1

Hi all,

I tried to declare one of my json fields (mmsi) as es.mapping.id in EsBolt but it crashes with EsHadoopIllegalArgumentException. How do I resolve the problem? My stack trace is as follows:

java.lang.RuntimeException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: [JsonExtractor for field [mmsi]] cannot extract value from entity [class java.lang.String] | instance [{"records":[{"receiveTime":1353801681000,"mmsi":111111111,"imo":7110024,"callSign":"VJHN","shipName":"TANUNDA","shipType":"Tug","toBow":8,"toStern":17,"toPort":4,"toStarboard":4,"epfd":1,"etaMonth":0,"etaDay":0,"etaHour":24,"etaMinute":60,"draught":3.0,"destination":"BRISBANE !","dte":false,"completeness":3}]}]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.daemon.executor$fn__3439$fn__3451$fn__3498.invoke(executor.clj:748) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463) ~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: [JsonExtractor for field [mmsi]] cannot extract value from entity [class java.lang.String] | instance [{"records":[{"receiveTime":1353801681000,"mmsi":111111111,"imo":7110024,"callSign":"VJHN","shipName":"TANUNDA","shipType":"Tug","toBow":8,"toStern":17,"toPort":4,"toStarboard":4,"epfd":1,"etaMonth":0,"etaDay":0,"etaHour":24,"etaMinute":60,"draught":3.0,"destination":"BRISBANE !","dte":false,"completeness":3}]}]
at org.elasticsearch.hadoop.serialization.bulk.AbstractBulkFactory$FieldWriter.write(AbstractBulkFactory.java:97) ~[elasticsearch-spark_2.11-2.1.0.Beta4.jar:2.1.0.Beta4]
at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.writeTemplate(TemplatedBulk.java:80) ~[elasticsearch-spark_2.11-2.1.0.Beta4.jar:2.1.0.Beta4]
at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.write(TemplatedBulk.java:56) ~[elasticsearch-spark_2.11-2.1.0.Beta4.jar:2.1.0.Beta4]
at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:148) ~[elasticsearch-spark_2.11-2.1.0.Beta4.jar:2.1.0.Beta4]
at org.elasticsearch.storm.EsBolt.execute(EsBolt.java:116) ~[elasticsearch-storm-2.1.0.Beta4.jar:2.1.0.Beta4]
at backtype.storm.daemon.executor$fn__3439$tuple_action_fn__3441.invoke(executor.clj:633) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.daemon.executor$mk_task_receiver$fn__3362.invoke(executor.clj:401) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.disruptor$clojure_handler$reify__1445.onEvent(disruptor.clj:58) ~[storm-core-0.9.4.jar:0.9.4]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.9.4.jar:0.9.4]
... 6 common frames omitted

Thanks,
CG

1 Like
2

Hi all,

I realize that esBolt cannot use any fields in a json array as the document id. It works if the field (mmsi) is not in an array. By the way, is it possible to declare mappings in esBolt? I can't find any documents on mappings using esBolt. Right now, I have to create the index with the correct mappings in Elasticsearch, then starts to populate Elasticsearch using my bolt which is like 2 separate processes.

Thanks,
CG