Logstash ECS Compatiblity Issues

[teamomni]

We have moved our Logstash indices to data stream recently but are having issues with the geopoints. As a result, the geofencing for everything including the Elastic agents are broken. At first there was the error:

Pipeline error {:pipeline_id=>"network_logs", :exception=>#<LogStash::ConfigurationError: GeoIP Filter in ECS-Compatiblity mode requires a target when source is not an ip sub-field, eg. [client][ip]>

But ECS compatibility has been disabled for the two IP variables while the rest is v8 compatible:

filter {
  if [source.ip] =~ /^192.168.1.*$/ {
    mutate {
      add_field => [ "source.geo.location", "41.12, -71.34" ]
    }
    geoip {
      source => "destination.ip"
      ecs_compatibility => disabled
    }
    mutate {
      add_field => [ "destination.geo.location", "%{[geoip][latitude]}, %{[geoip][longitude]}" ]
    }
  }
  if [destination.ip] =~ /^192.168.1.*$/ {
    mutate {
      add_field => [ "destination.geo.location", "41.12, -71.34" ]
    }
    geoip {
      source => "source.ip"
      ecs_compatibility => disabled
    }
    mutate {
      add_field => [ "source.geo.location", "%{[geoip][latitude]}, %{[geoip][longitude]}" ]
    }
  }
}

Is it possible to edit the filter so that the source.ip and destination.ip would be compliant with ECS without the ‘target’ error so that the map works properly again?

[Badger]

Do all your fields really have periods in their names? Should [destination.ip] be [destination][ip]? Likewise should source.geo.location be [source][geo][location]?

The default template for an Elasticsearch output makes [geoip][location] a geo_point. For any other field you will need a template that sets the type.

[teamomni]

Those fields do have periods rather than brackets, they have been that way for years, but I will test your way and see if that fixes the problem.

[teamomni]

Thank you @Badger , that did fix the ECS compatibility issue.

Another question, the agents are supplying POINT geo locations, but the logs coming in with the “lat,lon” strings are only keywords. Do I need to just reindex the entire logs-* or will the string remain only a keyword?

[Badger]

If you indexed data without a template that made those fields geo_points then you will need to reindex them with such a template.

[teamomni]

That's what I thought, thank you.