I have picked up a Logstash server to look after and I can see some incoming events from a Kinesis input that is failing to be indexed:
Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"fargate-container-development-2020.01.16", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x7677b8d7>], :response=>{"index"=>{"_index"=>"fargate-container-development-2020.01.16", "_type"=>"doc", "_id"=>"1234567890", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [time]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"For input string: \"2020-01-16T11:29:38.157Z\""}}}}}
I thought that should be a simple fix and tried this:
filter {
if "kinesis" in [tags] and "fargatecontainerlogs" in [tags] {
date {
match => [ "time", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "ISO8601" ]
target => "@timestamp"
remove_field => [ "time" ]
}
json {
source => "message"
remove_field => [ "message" ]
}
}
}
Any idea where I went wrong?