Filebeat cannot send data to logstash

[binschlag]

My filebeat - agents cannot send data to logstash anymore.

In logstash.log i can see
failed to parse field [agent] of type [text] in document with id

whole message:

[2020-07-16T15:04:41,302][WARN ][logstash.outputs.elasticsearch][main][ee59c3f2b510dd61712dbfff8e1f178035c6d7df420a50210b9ef6339748583e] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2020.07", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x3a143ad5>], :response=>{"index"=>{"_index"=>"logstash-2020.07", "_type"=>"_doc", "_id"=>"S7W6V3MBEXAfkiTnG5lP", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [agent] of type [text] in document with id 'S7W6V3MBEXAfkiTnG5lP'. Preview of field's value: '{hostname=btwebsrv002, name=btwebsrv002, id=11bcdaa9-a452-4e0a-8967-32979b38088c, type=filebeat, ephemeral_id=8153e32b-7c0a-4307-b731-ddc46bbbe586, version=7.8.0}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1068"}}}}}

The whole Elastic Stack is already version 7.8 (Elasticsearch, Kibana, Logstash, Filebeat on each hosts)

Today after restarting a filebeat service on one host, it stopped working and now no one of filebeat agents can send any data to elasticsearch through logstash.

I did a lot of troubleshooting but got stuck. How can i fix this issue?

I want to add, that everything is running on-premise.

[rugenl (Len Rugen)]

"agent" is likely a test field in the index, but now is an object with several fields, like agent.version (I think, from memory). This is usually handled by having the agent version in the index name, so when fields change, a new index starts.