I solved this. It was a Logstash config issue.
In the logs of ElasticSearch (see below) I noticed the exception was occurring on field input, which was precisely one of the fields I was expecting to see in my "disappearing" logs.
[2020-04-03T09:53:37,718][DEBUG][o.e.a.b.TransportShardBulkAction] [instance-42] [honeypot-logstash-2020.04.03-000001][0] failed to execute bulk item (index) index {[honeypot-logstash][_doc][mo91P3EBYpA6orC0z2C2], source[_na_]}
org.elasticsearch.index.mapper.MapperParsingException: object mapping for [input] tried to parse field [input] as object, but found a concrete value
Then, in Logstash, I noticed this type of log, with the same error message:
Mar 31 05:44:14 instance-42 logstash[22592]: [2020-03-31T03:44:14,883][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cowrie-logstash", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x2082e9af>], :response=>{"index"=>{"_index"=>"cowrie-logstash-2020.03.20-000001", "_type"=>"_doc", "_id"=>"Go-wLnEBYpA6orC0jiYT", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [input] tried to parse field [input] as object, but found a concrete value"}}}}
So, I had a look at my entries and found out that in some cases, I had input field with sublayers
"input" => {
"type" => "log"
},
and in other cases, directly a string.
"command" => "echo \"this is my test\""
I searched for the problem (in particular this thread), and found out that this is a situation where Logstash is lost at types: in the first case, it expect an object, and it does not like in the second case to find a string.
Solution: I need to either always to have an object, or always a string, but not mix.
The fix will depend on your log structure, in my case (cowrie), I was doing
json {
source => "message"
}
which would read the JSON input from field message and create log structure from that. In my case, that was overwriting several fields such as input. The solution is to specify a target, so that fields from the JSON input don't overwrite other fields, and for instance, they'd go in honeypot.input, not input.
json {
source => "message"
target => "honeypot"
}