Logstash ECS Compatiblity Issues

We have moved our Logstash indices to data stream recently but are having issues with the geopoints. As a result, the geofencing for everything including the Elastic agents are broken. At first there was the error:

Pipeline error {:pipeline_id=>"network_logs", :exception=>#<LogStash::ConfigurationError: GeoIP Filter in ECS-Compatiblity mode requires a target when source is not an ip sub-field, eg. [client][ip]>

But ECS compatibility has been disabled for the two IP variables while the rest is v8 compatible:

filter {
  if [source.ip] =~ /^192.168.1.*$/ {
    mutate {
      add_field => [ "source.geo.location", "41.12, -71.34" ]
    }
    geoip {
      source => "destination.ip"
      ecs_compatibility => disabled
    }
    mutate {
      add_field => [ "destination.geo.location", "%{[geoip][latitude]}, %{[geoip][longitude]}" ]
    }
  }
  if [destination.ip] =~ /^192.168.1.*$/ {
    mutate {
      add_field => [ "destination.geo.location", "41.12, -71.34" ]
    }
    geoip {
      source => "source.ip"
      ecs_compatibility => disabled
    }
    mutate {
      add_field => [ "source.geo.location", "%{[geoip][latitude]}, %{[geoip][longitude]}" ]
    }
  }
}

Is it possible to edit the filter so that the source.ip and destination.ip would be compliant with ECS without the ‘target’ error so that the map works properly again?

Do all your fields really have periods in their names? Should [destination.ip] be [destination][ip]? Likewise should source.geo.location be [source][geo][location]?

The default template for an Elasticsearch output makes [geoip][location] a geo_point. For any other field you will need a template that sets the type.

Those fields do have periods rather than brackets, they have been that way for years, but I will test your way and see if that fixes the problem.

Thank you @Badger , that did fix the ECS compatibility issue.

Another question, the agents are supplying POINT geo locations, but the logs coming in with the “lat,lon” strings are only keywords. Do I need to just reindex the entire logs-* or will the string remain only a keyword?

If you indexed data without a template that made those fields geo_points then you will need to reindex them with such a template.

That's what I thought, thank you.