GeoIp-problems

Elastic Stack Logstash
1

I'm trying to filter out the GeoIp from the log I'm ingesting into Elasticsearch.
The log pattern is like this:
{"@message":"Successful api request","@timestamp":"2018-01-22T08:24:00.162Z","@fields":{"origin":"193.xx.xx.xxx","environment":"production_beta","label":"askquestiongui","level":"info"}}

I've tested with this filter setting without any luck:
filter { geoip { source => "@fields.origin.keyword" } }

Output:
{ "@version" => "1", "http_poller_metadata" => { "request" => { "headers" => { "Accept" => "application/json", "x-lm-api" => "xxxxxxxxxxxxxxxxxxxxxxxxx" }, "method" => "get", "url" => "http://xx.xx.xx/xx/log" }, "response_headers" => { "date" => "Mon, 22 Jan 2018 11:21:05 GMT", "x-global-transaction-id" => "139098999", "last-modified" => "Mon, 22 Jan 2018 11:20:05 GMT", "x-backside-transport" => "OK OK", "transfer-encoding" => "chunked", "x-powered-by" => "Express", "connection" => "Keep-Alive", "content-type" => "text/plain; charset=UTF-8", "etag" => "W/\"5948-1611d98bf6a\"", "cache-control" => "public, max-age=0" }, "code" => 200, "response_message" => "OK", "times_retried" => 0, "runtime_seconds" => 5.095, "name" => "test", "host" => "elastic6" }, "@message" => "Successful api request", "@timestamp" => 2018-01-22T11:20:05.226Z, "@fields" => { "environment" => "production_beta", "label" => "askquestiongui", "level" => "info", "origin" => "193.xx.xx.xxx" }, "tags" => [ [0] "_geoip_lookup_failure" ] }

I'm using Elasticsearch 5.6.4 and ingest-geoip is installed.

2

This syntax @fields.origin.keyword is not correct for Logstash and looks more like Elasticsearch syntax?

That said, your object looks like this:

"@fields":{"origin":"193.xx.xx.xxx" }

To access the 'origin' value you will need to use a field reference, and it'll look like this:

filter {
  geoip {
    source => "[@fields][origin]"
  }
}
3

That did the trick! Thanks alot for the help! :slight_smile:

4

And another strange thing... When I get the coordinates on my map, they are way of from where they should be. "location" => "18.056,59.3247" should be in Sweden, but shows up in the Indian Ocean.

Filter:

filter
{
geoip {
source => "[@fields][origin]"
target => "geoip"
add_field => [ "[geoip][location2]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][location2]", "%{[geoip][latitude]}" ]
}
mutate {
replace => ["[geoip][location]", "%{[geoip][location2]}" ]
}
}

Output in index:

geoip.city_name: Gävle
geoip.continent_code: EU
geoip.country_code2: SE
geoip.country_code3: SE
geoip.country_name: Sweden
geoip.ip: xxx.xx.xxx.xxx
geoip.latitude: 60.667
geoip.location: 17.1667,60.6667
geoip.location2: 17.1667, 60.6667
geoip.longitude: 17.167
geoip.postal_code: 800 10
geoip.region_code: X
geoip.region_name: Gävleborg
geoip.timezone: Europe/Stockholm

5

Found the problem. Geoip had flipped long and lat for me. :wink:

6

It looks like you're trying to build a GeoJSON value [longitude, latitude], but the geoip plugin already provides this, so there's no need for you to build this by hand as you are doing and it stores it as the 'location' field in { "lat": ..., "lon": ... } value that Elasticsearch and Kibana both understand:

Inputting some random IP 193.1.2.3:

{
  "geoip"      => {
    "continent_code" => "EU",
    "location"       => {
      "lat" => 53.3472,
      "lon" => -6.2439
    },
    "country_code2"  => "IE",
    "latitude"       => 53.3472,
    "country_name"   => "Ireland",
    "country_code3"  => "IE",
    "ip"             => "193.1.2.3",
    "timezone"       => "Europe/Dublin",
    "longitude"      => -6.2439
  },
  "message"    => "193.1.2.3",
  "@version"   => "1",
  "@timestamp" => 2018-02-01T16:47:48.697Z,
  "host"       => "fancypants",
  "type"       => "stdin"
}
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.