Not able to filter on geoip.location?


(Andy) #1

So I'm fairly new to ELK Stack over the last few weeks, but I have most my systems reporting in fine for Windows logs, and Linux Secure/System logs OK. Today I added in the Apache logs for my DMZ and again these are getting their OK and being filtered correctly, so I tried adding GeoIP for lookups.

I can get GeoIP data showing up OK, but I can't do a Tile Map as I don't have a geo_point index type for the geoip.location? I followed some guides and in my filter I have this below which gives me a combined field for the long and lat, but it just comes in as a number, so I can't use it.:

geoip {
  source => "clientip"
  add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
  add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
}
mutate {
  convert => [ "[geoip][coordinates]", "float"]
} 

I've tried editing the template to include it, but template editing is totally new for me so I'm a bit out of my depth there so probably doing it wrong.

Any ideas how I can get a field to show up as a GeoPoint that I can use? Here is my JSON for a event with some details scrubbed for privacy. The external IP and coordinates are valid, I've just changed it for here.

{
  "_index": "filebeat-2017.08.10",
  "_type": "log",
  "_id": "AV3OWX9sLIizUbo1aUC3",
  "_version": 1,
  "_score": null,
  "_source": {
    "request": "/",
    "agent": "\"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.0 Mobile/14G60 Safari/602.1\"",
    "geoip": {
      "timezone": "Pacific/Auckland",
      "ip": "123.123.123.123",
      "latitude": -10.123,
      "coordinates": [
        456.2322,
        -10.123
      ],
      "continent_code": "OC",
      "city_name": "Auckland",
      "country_name": "New Zealand",
      "country_code2": "NZ",
      "country_code3": "NZ",
      "region_name": "Auckland",
      "location": {
        "lon": 456.2322,
        "lat": 10.123
      },
      "postal_code": "1150",
      "region_code": "AUK",
      "longitude": 456.2322
    },
    "offset": 66048,
    "auth": "-",
    "ident": "-",
    "input_type": "log",
    "verb": "GET",
    "source": "/var/log/httpd/access_log",
    "message": "123.123.123.123 - - [11/Aug/2017:10:52:46 +1200] \"GET / HTTP/1.1\" 403 3539 \"-\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.0 Mobile/14G60 Safari/602.1\"",
    "type": "log",
    "tags": [
      "Auckland NZ",
      "beats_input_codec_plain_applied"
    ],
    "referrer": "\"-\"",
    "@timestamp": "2017-08-10T22:52:46.000Z",
    "response": "403",
    "bytes": "3539",
    "clientip": "123.123.123.123",
    "@version": "1",
    "beat": {
      "hostname": "dmz1.external",
      "name": "dmz1.external",
      "version": "5.5.1"
    },
    "host": "dmz1.external",
    "httpversion": "1.1",
    "fields": {
      "logtype": "apache_access",
      "env": "Development"
    },
    "timestamp": "11/Aug/2017:10:52:46 +1200"
  },
  "fields": {
    "@timestamp": [
      1502405566000
    ]
  },
  "highlight": {
    "beat.name": [
      "@kibana-highlighted-field@dmz1.external@/kibana-highlighted-field@"
    ],
    "beat.hostname": [
      "@kibana-highlighted-field@dmz1.external@/kibana-highlighted-field@"
    ],
    "host": [
      "@kibana-highlighted-field@dmz1.external@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1502405566000
  ]
}

(Mark Walkom) #2

FYI we’ve renamed ELK to the Elastic Stack, otherwise beats feels left out :wink:

Have a look at https://www.elastic.co/blog/geoip-in-the-elastic-stack, basically all that config code is over complicating things and the blog post goes into how to resolve that. Let us know if you have other questions or problems though!


(Andy) #3

Thanks. I had followed that blog originally but got no joy, so tried the extra configs it mentioned and still nothing.

I can get all the GeoIP data in my indexes, it resolves everything OK and I get no errors, I just can't get the geo_point field created. I've imported the new templates but still nothing get generated for it.

I have been able to create bar charts for city/country OK, just not the pretty maps :slight_smile:


(Mark Walkom) #4

What does the mapping look like then?


(Andy) #5

So this is what I see for a typical event that happened at the weekend. In the JSON for the entry I have this.

"geoip": {
  "timezone": "Pacific/Auckland",
  "ip": "125.239.158.17",
  "latitude": -36.7932,
  "continent_code": "OC",
  "city_name": "Auckland",
  "country_name": "New Zealand",
  "country_code2": "NZ",
  "country_code3": "NZ",
  "region_name": "Auckland",
  "location": {
    "lon": 175.0362,
    "lat": -36.7932
  },
  "postal_code": "1081",
  "region_code": "AUK",
  "longitude": 175.0362

But that location field, doesn't show up as a geo_point object that I can use in a map. I had already imported all the default winlogbeat and filebeat templates before I got the data, so today I added that other template below.

{
  "order": 1,
  "template": "winlogbeat-*",
  "mappings": {
    "_default_": {
      "properties": {
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "float" },
            "longitude" : { "type" : "float" }
          }
        }
      }
    }
  }
}

I then tried to generate some more traffic, and again it appears the same, geo-ip resolves the location but there is no geo_point index I can search from.


(Mark Walkom) #6

You have the correct geoip.location mapping, but if you look at your field you have geoip.location.lat and geoip.location.lon.

Just do;

geoip {
  source => "clientip"
}

And you should be fine.

Also I can see your mapping is for winlogbeat, but your original message is for filebeat?


(Andy) #7

I actually already have the below in my beats config.

Syslog filter:

	   geoip {
	      source => "[system][auth][ssh][ip]"
	      target => "[system][auth][ssh][geoip]"
   	}

Apache filter:

   	geoip {
	        source => "clientip"
    	}

Everything else filter:

	geoip {
	source => "[event_data][IpAddress]"
    }

The geoip.location mapping is showing up but not as a geo_point so I can't use it for maps.


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.