filbeat: 6.2.2 (arm), libbeat 6.2.2
logstash: 6.2.2
Filebeat after 'some time' ended up logging:
2018-02-25T13:37:42.058Z ERROR logstash/async.go:235 Failed to publish events caused by: read tcp 192.168.1.116:42208->192.168.1.142:5044: read: connection reset by peer
Logstash 5044 port was present. Restarting Logstash seemed the only way to unblock things and while it attempted to restart it was hung waiting for the mutate filter to remove some fields. Many of these lines were observed in logstash log:
2018-02-25T08:39:07,056][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{["LogStash::Filters::Mutate", {"remove_field"=>["source", "host", "ocp_authHash", "message", "timestamp"], "id"=>"26a13f4ef0f4def3b587eee4175701a05c46566f68eef329901ef422bb29050a"}]=>[{"thread_id"=>34, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:90:in `read_batch'"}]}}
The net effect was Filebeat could not send any logs to Logstash and Logstash didn't have anything in it's logs say it was jammed up until I attempted to restart it.
I've seen many similar titled (expired) discussions on this site about very similar experiences throughout 2017 and 2018. I opened a new topic because those appeared to have expired without resolution/workaround.
Is this a known problem with mutate filter, or something else in a simple installation of single instances of elasticsearch, logstash and filebeat? Filebeat has a 4 of prospectors, 2 works. Logstash configuration is default.