Not able to filter on geoip.location?

Elastic Stack Kibana
1

So I'm fairly new to ELK Stack over the last few weeks, but I have most my systems reporting in fine for Windows logs, and Linux Secure/System logs OK. Today I added in the Apache logs for my DMZ and again these are getting their OK and being filtered correctly, so I tried adding GeoIP for lookups.

I can get GeoIP data showing up OK, but I can't do a Tile Map as I don't have a geo_point index type for the geoip.location? I followed some guides and in my filter I have this below which gives me a combined field for the long and lat, but it just comes in as a number, so I can't use it.:

geoip {
  source => "clientip"
  add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
  add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
}
mutate {
  convert => [ "[geoip][coordinates]", "float"]
}

I've tried editing the template to include it, but template editing is totally new for me so I'm a bit out of my depth there so probably doing it wrong.

Any ideas how I can get a field to show up as a GeoPoint that I can use? Here is my JSON for a event with some details scrubbed for privacy. The external IP and coordinates are valid, I've just changed it for here.

{
  "_index": "filebeat-2017.08.10",
  "_type": "log",
  "_id": "AV3OWX9sLIizUbo1aUC3",
  "_version": 1,
  "_score": null,
  "_source": {
    "request": "/",
    "agent": "\"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.0 Mobile/14G60 Safari/602.1\"",
    "geoip": {
      "timezone": "Pacific/Auckland",
      "ip": "123.123.123.123",
      "latitude": -10.123,
      "coordinates": [
        456.2322,
        -10.123
      ],
      "continent_code": "OC",
      "city_name": "Auckland",
      "country_name": "New Zealand",
      "country_code2": "NZ",
      "country_code3": "NZ",
      "region_name": "Auckland",
      "location": {
        "lon": 456.2322,
        "lat": 10.123
      },
      "postal_code": "1150",
      "region_code": "AUK",
      "longitude": 456.2322
    },
    "offset": 66048,
    "auth": "-",
    "ident": "-",
    "input_type": "log",
    "verb": "GET",
    "source": "/var/log/httpd/access_log",
    "message": "123.123.123.123 - - [11/Aug/2017:10:52:46 +1200] \"GET / HTTP/1.1\" 403 3539 \"-\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.0 Mobile/14G60 Safari/602.1\"",
    "type": "log",
    "tags": [
      "Auckland NZ",
      "beats_input_codec_plain_applied"
    ],
    "referrer": "\"-\"",
    "@timestamp": "2017-08-10T22:52:46.000Z",
    "response": "403",
    "bytes": "3539",
    "clientip": "123.123.123.123",
    "@version": "1",
    "beat": {
      "hostname": "dmz1.external",
      "name": "dmz1.external",
      "version": "5.5.1"
    },
    "host": "dmz1.external",
    "httpversion": "1.1",
    "fields": {
      "logtype": "apache_access",
      "env": "Development"
    },
    "timestamp": "11/Aug/2017:10:52:46 +1200"
  },
  "fields": {
    "@timestamp": [
      1502405566000
    ]
  },
  "highlight": {
    "beat.name": [
      "@kibana-highlighted-field@dmz1.external@/kibana-highlighted-field@"
    ],
    "beat.hostname": [
      "@kibana-highlighted-field@dmz1.external@/kibana-highlighted-field@"
    ],
    "host": [
      "@kibana-highlighted-field@dmz1.external@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1502405566000
  ]
}
2

FYI we’ve renamed ELK to the Elastic Stack, otherwise beats feels left out :wink:

Have a look at https://www.elastic.co/blog/geoip-in-the-elastic-stack, basically all that config code is over complicating things and the blog post goes into how to resolve that. Let us know if you have other questions or problems though!

3

Thanks. I had followed that blog originally but got no joy, so tried the extra configs it mentioned and still nothing.

I can get all the GeoIP data in my indexes, it resolves everything OK and I get no errors, I just can't get the geo_point field created. I've imported the new templates but still nothing get generated for it.

I have been able to create bar charts for city/country OK, just not the pretty maps :slight_smile:

4

What does the mapping look like then?

5

So this is what I see for a typical event that happened at the weekend. In the JSON for the entry I have this.

"geoip": {
  "timezone": "Pacific/Auckland",
  "ip": "125.239.158.17",
  "latitude": -36.7932,
  "continent_code": "OC",
  "city_name": "Auckland",
  "country_name": "New Zealand",
  "country_code2": "NZ",
  "country_code3": "NZ",
  "region_name": "Auckland",
  "location": {
    "lon": 175.0362,
    "lat": -36.7932
  },
  "postal_code": "1081",
  "region_code": "AUK",
  "longitude": 175.0362

But that location field, doesn't show up as a geo_point object that I can use in a map. I had already imported all the default winlogbeat and filebeat templates before I got the data, so today I added that other template below.

{
  "order": 1,
  "template": "winlogbeat-*",
  "mappings": {
    "_default_": {
      "properties": {
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "float" },
            "longitude" : { "type" : "float" }
          }
        }
      }
    }
  }
}

I then tried to generate some more traffic, and again it appears the same, geo-ip resolves the location but there is no geo_point index I can search from.

6

You have the correct geoip.location mapping, but if you look at your field you have geoip.location.lat and geoip.location.lon.

Just do;

geoip {
  source => "clientip"
}

And you should be fine.

Also I can see your mapping is for winlogbeat, but your original message is for filebeat?

7

I actually already have the below in my beats config.

Syslog filter:

	   geoip {
	      source => "[system][auth][ssh][ip]"
	      target => "[system][auth][ssh][geoip]"
   	}

Apache filter:

   	geoip {
	        source => "clientip"
    	}

Everything else filter:

	geoip {
	source => "[event_data][IpAddress]"
    }

The geoip.location mapping is showing up but not as a geo_point so I can't use it for maps.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.