Hello, I am using filebeat to collect Diagnostic Logs for Entra ID on Azure.
I found that the properties.location field has different types in different logs. In some logs it should be configured as object type, in other fields it should be configured as keyword type.
Currently I have the mapping of this field set to object, so if the properties.location field should be of type keyword in the logs it won't write to ES properly.
The error log is captured as follows:
Could not index event to Elasticsearch.
......
response=>{“create”=>{“status”=>400, “error”=>{“type”=>“document_parsing_exception”, “reason”=>“[1:1369] failed to parse field [ properties.location.geoCoordinate”, “reason” => “[1:1369] failed to parse field [ properties.location.geoCoordinates] of type [geo_point]“, ‘caused_by’=>{”type“=>”parse_exception“, ‘reason’=>”field must be either lat/lon, geohash string or type/coordinates”
I want to process the logs sent by filebeat in logstash, how do I write the filter in logstash? Logstash version is 8.14.3. Thanks.