My field `input` is modified, and some events are filtered (against my will ;-))

Elastic Stack Logstash
2

I fixed this. It was a Logstash config issue. In Logstash, I noticed this type of log, with the same error message:

Mar 31 05:44:14 instance-42 logstash[22592]: [2020-03-31T03:44:14,883][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cowrie-logstash", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x2082e9af>], :response=>{"index"=>{"_index"=>"cowrie-logstash-2020.03.20-000001", "_type"=>"_doc", "_id"=>"Go-wLnEBYpA6orC0jiYT", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [input] tried to parse field [input] as object, but found a concrete value"}}}}

So, I had a look at my entries and found out that in some cases, I had input field with sublayers

"input" => {
        "type" => "log"
    },

and in other cases, directly a string.

"command" => "echo \"this is my test\""

I searched for the problem (in particular this thread), and found out that this is a situation where Logstash is lost at types: in the first case, it expect an object, and it does not like in the second case to find a string.

Solution: I need to either always to have an object, or always a string, but not mix.

The fix will depend on your log structure, in my case (cowrie), I was doing

json {
 source => "message"
}

which would read the JSON input from field message and create log structure from that. In my case, that was overwriting several fields such as input. The solution is to specify a target, so that fields from the JSON input don't overwrite other fields, and for instance, they'd go in honeypot.input, not input.

json {
 source => "message"
 target => "honeypot"
}