Elastic search timeout exception for bulk execute (Index creation)

Elastic Stack Elasticsearch
1

Elastic search couldn't create a bulk item (index), It throws an error "Failed to acknowledge mapping update within [30s] ... " I don't why the timeout occurs. The error message is,

    [2019-01-31T08:20:17,547][DEBUG][o.e.a.b.TransportShardBulkAction] [filebeat-2019.01.31][2] failed to execute bulk item (index) BulkShardRequest [[filebeat-2019.01.31][2]] containing [index {[filebeat-2019.01.31][doc][R54jpWgBz-pKjtWo2_zo], source[{"offset":13379306,"@timestamp":"2019-01-31T18:20:12.000Z","system.auth.timestamp":"Jan 31 08:20:12","beat":{"hostname":"elk-master","version":"6.4.2","name":"elk-master"},"tags":["beats_input_codec_plain_applied","SUCCESS_N1","success","_grokparsefailure"],"host":{"name":"elk-master"},"username":"test","system.auth.hostname":"elk-master","sourceAddress":"113.96.62.112","sourcePort":7518,"sys_message":"Failed password for invalid user","geoip":{"city_name":"Guangzhou","location":{"lon":113.25,"lat":23.1167},"country_code3":"CN","latitude":23.1167,"region_code":"44","ip":"113.96.62.112","country_code2":"CN","longitude":113.25,"continent_code":"AS","country_name":"China","region_name":"Guangdong","timezone":"Asia/Shanghai"},"input":{"type":"log"},"message":"Jan 31 08:20:12 elk-master sshd[32023]: Failed password for invalid user test from 113.96.62.112 port 7518 ssh2","source":"/var/log/auth.log","prospector":{"type":"log"},"program":"sshd","@version":"1","PID":"32023","service":"ssh2"}]}]
org.elasticsearch.ElasticsearchTimeoutException: Failed to acknowledge mapping update within [30s]
	at org.elasticsearch.cluster.action.index.MappingUpdatedAction.updateMappingOnMaster(MappingUpdatedAction.java:90) ~[elasticsearch-6.4.2.jar:6.4.2]
	at org.elasticsearch.cluster.action.index.MappingUpdatedAction.updateMappingOnMaster(MappingUpdatedAction.java:80) ~[elasticsearch-6.4.2.jar:6.4.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction$ConcreteMappingUpdatePerformer.updateMappings(TransportShardBulkAction.java:612) ~[elasticsearch-6.4.2.jar:6.4.2]
	………
	………
	………
	………
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.2.jar:6.4.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.2.jar:6.4.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

The logstash starts pretty well but fails when creating a pipeline with elasticsearch and the error it throw is,
[2018-11-25T21:55:18,860][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 24, column 3 (byte 882) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:incompile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:309:in block in converge_state'"]}

The memory allocation of elastic search is,

10.1.1.67: /dev/sda1        63G   41G   23G  65% /
10.1.1.62: /dev/sda1        63G   63G     0 100% /
10.1.1.71: /dev/sda1        63G   61G  1.4G  98% /
10.1.1.65: /dev/sdb        127G   42G   79G  35% /mnt/sdb #master
10.1.1.83: /dev/sda1       125G   89G   37G  72% /
10.1.1.88: /dev/sda1       125G   78G   48G  63% /

and free -m details are,
total used free shared buff/cache available
10.1.1.67: Mem: 16047 13298 246 180 2502 2200
10.1.1.62: Mem: 16046 13262 249 68 2534 2354
10.1.1.71: Mem: 16047 13326 326 180 2394 2172
10.1.1.65: Mem: 16046 8335 762 160 6949 7177
10.1.1.83: Mem: 30158 13551 406 223 16200 15951
10.1.1.88: Mem: 30158 13545 373 220 16239 15803

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.