An error while bulk indexing

[Ted_Jung]

Asking for an advice...what meaning below messgaes have.
Thanks in advance.

[2016-12-19 05:13:46,783][DEBUG][action.bulk ] [doctest02] [filebeat-2016.12.19][1] failed to execute bulk item (index) index {[filebeat-2016.12.19][apache][AVkVfin1zSujsQlPvAXF], source[{"message":"XX.XX.XX.XX - - [19/Dec/2016:03:18:41 +0000] "GET /cc/editor.js HTTP/1.1" 200 366349","@version":"1","@timestamp":"2016-12-19T03:18:41.000Z","input_type":"log","tenant":"tenant1","service":"service2","source":"/cdata/apache/ssl_access.log.20161219","type":"apache_access","count":1,"beat":{"hostname":"cl-was","name":"cl-was"},"offset":1063,"host":"cl-was","tags":["beats_input_codec_plain_applied"],"clientip":"XX.XX.XX.XX","ident":"-","auth":"-","timestamp":"19/Dec/2016:03:18:41 +0000","verb":"GET","request":"/cc/editor.js","httpversion":"1.1","response":200,"bytes":366349}]}
java.util.concurrent.TimeoutException: Failed to acknowledge mapping update within [30s]
at org.elasticsearch.cluster.action.index.MappingUpdatedAction.updateMappingOnMasterSynchronously(MappingUpdatedAction.java:122)
at org.elasticsearch.cluster.action.index.MappingUpdatedAction.updateMappingOnMasterSynchronously(MappingUpdatedAction.java:112)
at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:228)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:327)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:120)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:657)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:287)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

[Mark_Harwood]

It's a timeout waiting for the master to acknowledge a change to the mappings.

This could be a couple of things:

1. Do you have dedicated master nodes as per the recommendations? [1] A node that is double up as a data node can get overloaded and become unresponsive.
2. Does your incoming data introduce many new fields? e.g. if you have a timestamp as a field name (crazy, I know but I've seen it...) then every new doc will introduce a new mapping update if you have dynamic mappings.

Cheers
Mark

[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#master-node