GeoIP location has added brackets and not able to visualize

Elastic Stack Logstash
1

Hi, I have been trying to figure this out without success and hope there is someone out there that can assist with this matter. I have the GoeIP filter on the source and destination IP address and all works very well with country, etc. information. The challenge is with the GeoIP location information, it is presented in Kibana between [] and I am not able to select the defined filed also after converting. So it might be the [] that is causing the challenge or I have something incorrect with the filter, here is the filter information:
geoip { source => "source-address" target => "GeoIP-scr" database => "/opt/GeoIP/GeoLiteCity.dat" add_field => [ "[GeoIP-scr][coordinates]", "%{[GeoIP-scr][longitude]}" ] add_field => [ "[GeoIP-scr][coordinates]", "%{[GeoIP-scr][latitude]}" ] } mutate { convert => [ "[GeoIP-scr][coordinates]", "float"] }

In kibana it is presented a:
GeoIP-scr.coordinates [28.05,-26.205]

Any assistance, would be truly appreciated.

1 Like
2

Did you map that field as a geopoint in ES before indexing?

3

HI Warkolm, no how do I do this?

4

Check the existing LS template in ES via the _templates endpoint, you want to adapt the geoip field it has.

5

When querying the templates, I get the flowing:

[root@localhost ~]# curl 'localhost:9200/_template/?pretty>' {"logstash":{"order":0,"template":"logstash-*","settings":{"index.refresh_interval":"5s"},"mappings":{"_default_":{"dynamic_templates":[{"message_field":{"mapping":{"index":"analyzed","omit_norms":true,"type":"string"},"match_mapping_type":"string","match":"message"}},{"string_fields":{"mapping":{"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"ignore_above":256,"index":"not_analyzed","type":"string"}}},"match_mapping_type":"string","match":"*"}}],"_all":{"omit_norms":true,"enabled":true},"properties":{"geoip":{"dynamic":true,"type":"object","properties":{"location":{"type":"geo_point"}}},"@version":{"index":"not_analyzed","type":"string"}}}},"aliases":{}}}[root@localhost ~]#

What exactly must I change to be compatible?

6

With your current Logstash template, you can use this config in logstash to get geoip data

geoip {
		source => "source-address"
		target => "geoip"
		fields => ["country_code2", "country_name", "continent_code", "region_name", "real_region_name", "city_name", "postal_code", "timezone", "location"]
	}

I used fields to limit the number of fields exported to ES. Remove it if you want all fields under geoip. For Kibana to graph geoip location, you have to use geo_point data type for location/coodinates field, but your current logstash config is sending float data type.

7

Thank you Anhlqn for the advise. I tried to convert the coordinates to geo_point and get the following error in logstash:
Error: Cannot register filter mutate plugin. The error reported is: Invalid conversion type 'geo_point', expected one of 'string,integer,float,boolean' You may be interested in the '--configtest' flag which you can use to validate logstash's configuration before you choose to restart a running system.
It seems the geo_point option is not available, unless I am missing something?

8

Provide your config please!

9

Don't do any conversion with mutate plugin. geoip plugin automatically outputs the right data format. Just remove all the filters you have in logstash right now and use the config I gave you. It would be more helpful if you post your whole logstash config.

1 Like
10

I have the same issue - running Logstash 2.3.1 with the following GeoIP config:

if [clientip] { geoip { source => "clientip" target => "geoip" fields => [ "city_name", "continent_code", "country_code2", "country_name", "location", "region_name" ] } }

output is via Gelf into Graylog (currently running v2.0.0 b3) against ElasticSearch 2.1.

When looking at Graylog, the GeoIP data in the message is:

geoip_continent_code: AF geoip_country_code2: ZA geoip_country_name: South Africa geoip_location: [24.0,-29.0]

Is the above related to this: https://github.com/logstash-plugins/logstash-filter-geoip/issues/61 ?

11

Hi, is there a option how to remove [] in GeoIP location field information that it is presented in Kibana?