Logstash Giving failed to parse [Records.apiVersion]

[Suresh_Pal]

Hi Team,

Great morning, Hope you are doing awesome.

I am trying to ship cloudtrail logs in my ELK. But I'm getting following warning

[WARN ] 2018-11-14 23:45:37.775 [Ruby-0-Thread-6@[main]>worker0: :1] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.11.15", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x52da2cb2], :response=>{"index"=>{"_index"=>"cloudtrail-2018.11.15", "_type"=>"doc", "_id"=>"LrWwFWcBV6thQHuuWGBp", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [Records.apiVersion]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "2017_10_30" is malformed at "_10_30""}}}}}

Though logs are shipping in my kibana. but I saw a warning message on kibana UI: "Objects in arrays are not well supported"

My config file is as below :

input {
s3 {
bucket => "abc"
prefix => "abcded/"
region => "us-east-1"
type => "cloudtrail"
codec => json_lines
secret_access_key => "asdafafafas"
access_key_id => "asdfsafsfsd"
}
}
filter {
if [type] == "cloudtrail" {
geoip {
source => "sourceIPAddress"
target => "geoip"
add_tag => [ "cloudtrail-geoip" ]
}
}
mutate {
add_field => { "indexname" => "cloudtrail-%{+YYYY.MM.dd}" }
}
mutate {
lowercase => [ "indexname" ]
}

mutate {
  gsub => [
    "eventSource", "\.amazonaws\.com$", "",
    "apiVersion", "_", "-"
  ]
}

}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "cloudtrail-%{+YYYY.MM.dd}"
}
}

Note : logstash.version"=>"6.3.1
elasticsearch version 6.3.2

[Suresh_Pal]

Can anyone please help me out?

[Eniqmatic (Lewis Barclay)]

Does not look like your gsub is working correctly. Interestingly, the field in the error says "Records.apiVersion" but in your gsub you are accessing "apiVersion" is this correct??

[Suresh_Pal]

Even using Records.apiVersion in gsub it giving the same error.
Although apiVersion in gsub is correct way.

[Eniqmatic (Lewis Barclay)]

So what is the field name? Records.apiVersion or apiVersion?

[Suresh_Pal]

apiVersion,

[Suresh_Pal]

Now i'm getting these error as well .
[WARN ] 2018-11-30 01:10:28.502 [Ruby-0-Thread-9@[main]>worker3: :1] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.11.30", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x4dd97c5], :response=>{"index"=>{"_index"=>"cloudtrail-2018.11.30", "_type"=>"doc", "_id"=>"iKM9Y2cB4Qlyba7XPFU2", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [cloudtrail-2018.11.30] as the final mapping would have more than 1 type: [apiVersion_type, doc]"}}}}
[WARN ] 2018-11-30 01:10:28.502 [Ruby-0-Thread-9@[main]>worker3: :1] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.11.30", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x5c6f18b5], :response=>{"index"=>{"_index"=>"cloudtrail-2018.11.30", "_type"=>"doc", "_id"=>"iaM9Y2cB4Qlyba7XPFU2", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [Records.apiVersion] of different type, current_type [text], merged_type [date]"}}}}