Geoip stopped consolidating coordinates after logstash upgrade to 8.0

Elastic Stack Logstash
1

My logstash for geo tagging IPs are failing after logstash upgrade 8.0.

Here's a snippet , i get the lat and lon values , but cant seem to get into the "destlocation" field


         "Dest_IP" => {
        "geo" => {
                "country_name" => "United States",
                    "location" => {
                "lat" => 37.751,
                "lon" => -97.822
            },
                    "timezone" => "America/Chicago",
            "country_iso_code" => "US",
              "continent_code" => "NA"
        },
         "ip" => "147.160.184.40"

                 "@timestamp" => 2022-02-23T10:10:11.626Z,
               "destlocation" => [
        [0] 0.0,
        [1] 0.0

code

 geoip
        {
        target => "Dest_IP"
        source => "Forward_IP"
        tag_on_failure => ["failed!"]
        }
        }
        mutate
                {
                add_field => ["sourcelocation","%{[SRC_IP][longitude]}","tmplat","%{[SRC_IP][latitude]}"]
                add_field => ["destlocation","%{[Dest_IP][longitude]}","tmplatdst","%{[Dest_IP][latitude]}"]
                }
        mutate
                {
                        merge => ["sourcelocation","tmplat"]
                        merge => ["destlocation","tmplatdst"]
                }
        mutate
                {
                        convert => [ "sourcelocation","float"]
                        convert => ["destlocation","float"]
                }
        mutate
                {
                        remove_field => [ "tmplat","tmplatdst"]
                }

when starting up my config, i see lines about ECS, i think they may have something to do with it

[2022-02-23T16:55:03,135][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-02-23T16:55:03,143][INFO ][logstash.filters.kv      ][main] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2022-02-23T16:55:03,151][WARN ][logstash.filters.geoip   ][main] ECS expect `target` value `Dest_IP` in ["client", "destination", "host", "observer", "server", "source"]
[2022-02-23T16:55:08,601][ERROR][logstash.filters.geoip.databasemanager] Connect to geoip.elastic.co:443 [geoip.elastic.co/104.154.207.153] failed: connect timed out {:cause=>org.apache.http.conn.ConnectTimeoutException: Connect to geoip.elastic.co:443 [geoip.elastic.co/104.154.207.153] failed: connect timed out}
[2022-02-23T16:55:08,672][INFO ][logstash.filters.geoip   ][main] Using geoip database {:path=>"/var/lib/logstash/plugins/filters/geoip/CC/GeoLite2-City.mmdb"}
[2022-02-23T16:55:08,697][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2022-02-23T16:55:08,783][WARN ][logstash.filters.geoip   ][main] ECS expect `target` value `SRC_IP` in ["client", "destination", "host", "observer", "server", "source"]
[2022-02-23T16:55:08,786][INFO ][logstash.filters.geoip   ][main] Using geoip database {:path=>"/var/lib/logstash/plugins/filters/geoip/CC/GeoLite2-City.mmdb"}
[2022-02-23T16:55:08,787][WARN ][logstash.filters.geoip   ][main] ECS expect `target` value `Dest_IP` in ["client", "destination", "host", "observer", "server", "source"]
[2022-02-23T16:55:08,788][INFO ][logstash.filters.geoip   ][main] Using geoip database {:path=>"/var/lib/logstash/plugins/filters/geoip/CC/GeoLite2-City.mmdb"}
[2022-02-23T16:55:08,789][WARN ][logstash.filters.geoip   ][main] ECS expect `target` value `SRC_IP` in ["client", "destination", "host", "observer", "server", "source"]
[2022-02-23T16:55:08,789][INFO ][logstash.filters.geoip   ][main] Using geoip database {:path=>"/var/lib/logstash/plugins/filters/geoip/CC/GeoLite2-City.mmdb"}
[2022-02-23T16:55:08,847][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/conf.d/tests/logstash-dissect_f_line_test.conf"], :thread=>"#<Thread:0x1e7b7fe1 run>"}
2

Try to disable the ecs_compatibility for that pipeline according to the documentation.

Use pipeline.ecs_compatibility: disabled in the pipelines.yml for each pipeline or in the logstash.yml for all the pipelines.

It seems that the geoip information is being created in the field Dest_IP.geo.location.lat and Dest_IP.geo.location.lon, but your mutate filter is making a reference to another field, Dest_IP.longitude, which does not seems to exist.

1 Like
3

thank you for the help , this worked :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.