GeoIP Filter in ECS-Compatiblity mode requires a `target` when `source` is not an `ip` sub-field, eg. [client][ip]

[e-ferrari (Bernd Lentes)]

Hi,

i'm trying to setup
Parsing Logs with Logstash | Logstash Reference [8.11] | Elastic.
But i get errors:

[2024-01-04T00:06:45,246][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: GeoIP Filter in ECS-Compatiblity
mode requires a `target` when `source` is not an `ip` sub-field, eg. [client][ip]>

My pipeline looks like this:

  beats {
    ssl => "false"
    port => "5044"
  }
}

filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}

output {
  stdout { codec => rubydebug }
}

When i comment out the geoip filter i get this:

{
         "input" => {
        "type" => "log"
    },
           "ecs" => {
        "version" => "8.0.0"
    },
      "@version" => "1",
           "log" => {
        "offset" => 34227,
          "file" => {
            "path" => "/root/logstash-tutorial.log"
        }
    },
          "host" => {
        "name" => "suricata"
    },
         "agent" => {
        "ephemeral_id" => "601bd8bf-5443-410f-8712-56e85a58295e",
                "name" => "suricata",
             "version" => "8.11.3",
                  "id" => "d64a8bc7-2c9f-4ac7-a499-7daa78227552",
                "type" => "filebeat"
    },
        "source" => {
        "address" => "86.1.76.62"
    },
     "timestamp" => "04/Jan/2015:05:30:37 +0000",
    "@timestamp" => 2024-01-03T19:51:34.436Z,
          "http" => {
        "response" => {
            "status_code" => 200,
                   "body" => {
                "bytes" => 4877
            }
        },
         "version" => "1.1",
         "request" => {
            "referrer" => "http://www.semicomplete.com/projects/xdotool/",
              "method" => "GET"
        }
    },
           "url" => {
        "original" => "/style2.css"
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
    "user_agent" => {
        "original" => "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"
    },
       "message" => "86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] \"GET /style2.css HTTP/1.1\" 200 4877 \"http://www.semicomplete.com/projects/xdotool/\" \"Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0\"",
         "event" => {
        "original" => "86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] \"GET /style2.css HTTP/1.1\" 200 4877 \"http://www.semicomplete.com/projects/xdotool/\" \"Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0\""
    }
}

I don't see "geoip". Is this the problem?
I played around with the filter and tried:

filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "[source][address]"
    }
}

but didn't suceed.

Bernd

[Badger]

If your geoip filter has source that looks like [someField][ip], then it assumes that it is OK to add all the geoip fields under [someField]. Otherwise it needs you to tell it where to add those fields. You will also need an index template if you want to end up with a geo_point in an ES index.

If ECS is disabled then the target will be [geoip], which limits you to one geoip filter, which is not good, since a lot of people are dealing with logs that have variable sources and destinations.

[e-ferrari (Bernd Lentes)]

Hi,
this is now my pipeline:

    grok {
         match => { "message" => "%{COMBINEDAPACHELOG}"}
     }
      geoip {
         source => "[source][address]"
         target => "source"
     }
}

I get the following output:

        "address" => "86.1.76.62",
            "geo" => {
            "country_iso_code" => "GB",
                    "timezone" => "Europe/London",
                "country_name" => "United Kingdom",
                   "city_name" => "Burnley",
                 "region_name" => "Lancashire",
                 "postal_code" => "BB10",
             "region_iso_code" => "GB-LAN",
                    "location" => {
                "lon" => -2.2372,
                "lat" => 53.8265
            },

For me that looks fine.
What do you mean by "index template" ?

Thanks.

Bernd

[Badger]

@e-ferrari

What do you mean by "index template" ?

They are a way to tell elasticsearch how to interpret the data that you write to it.