Log source with different field type - mapper_parsing_exception

[nitzang (Nitzan Gotlib)]

Hello,

I recently started getting AWS cloud logs to ES and received many exceptions for indexing issues, such as:

[2020-02-03T15:35:56,728][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2020.02.03", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x7a78e3e4>], :response=>{"index"=>{"_index"=>"cloudtrail-2020.02.03", "_type"=>"_doc", "_id"=>"G-axC3AB9zZoOkkBpkYb", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [Records.requestParameters.DescribeVpcEndpointsRequest] tried to parse field [DescribeVpcEndpointsRequest] as object, but found a concrete value"}}}}

I did some reading and I know it's due to trying to get text data to object and vice-versa.

The problem is that's how Cloudtrail logs look like.

Is there any way to get both? or manipulating the logs (I'm using Logstash) that whenever a text field will arrive it will replace its name?

Any ideas?

Thanks