How to use geoip logstash filter?


(DharaniKumar) #1

In my log file i've lat and lon values in separate fields. I cant able to use tile map in kibana.

I have these details in my log file. Can some one guide me how to use it in to tilemap in kibana.

Thanks in advance.


(Magnus Bäck) #2

In short you have to

  • make sure the mapping of your index configures a designated field as a geo_point field (this is typically done via an index template), and
  • store your lat and lon values in that field in one of the formats that ES will recognize as geo_point (this is described in the ES documentation for geo_point).

(DharaniKumar) #3

"geoip": {
"dynamic": "true",
"properties": {
"location": {
"type": "geo_point"
}

this is my index template @magnusbaeck.


(Magnus Bäck) #4

What do the actual mappings of your index look like? What does an example event look like (copy/paste from Kibana's JSON tab in the Discover panel, no screenshots please)?


(DharaniKumar) #5

Do we need to download any database and place it in a specified path or kibana itself locate the lat and longi fields?


(Magnus Bäck) #6

Kibana will locate the fields by itself. Just make sure the index mappings are correct.


(DharaniKumar) #7

yeah @magnusbaeck. What about the database?
Do i need to download it from my side or its inbuilt in kibana(5.5.0).
I'm using logstash 5.5.0 too.


(Magnus Bäck) #8

What database are you talking about?


(DharaniKumar) #9

GeoLite2 database?!?!?! A city database to locate geolocation.


(Magnus Bäck) #10

As documented the geoip filter ships with a GeoIP database so you don't have to do anything about it, although the bundled database might be out of date. The documentation contains a paragraph or two about it.


(DharaniKumar) #11

"passcodeEnabled": "false",
"currentTimestamp": "2017-08-04 15:52:00.449",
"name": "Micromax-CBE",
"location": {
"latitude": "11.0130009",
"longitude": "76.9829831"
},

These are some datas that i inserted in location field. Lat and long fields as string.


(DharaniKumar) #12

but kibana does not recognise it as geopoints internally.


(Magnus Bäck) #13

According to your index template geoip.location is the name of your geo_point field.


(DharaniKumar) #14

Yeah @magnusbaeck i did set location to geoip.location like as follows:
PUT /filebeat-*/?pretty
{
"mappings": {
"syslog": {
"properties": {
"geoip.location": {
"type": "geo_point"
}
}
}
}
}

and i got this error:
{
"error": {
"root_cause": [
{
"type": "invalid_index_name_exception",
"reason": "Invalid index name [filebeat-], must not contain the following characters [ , ", , \, <, |, ,, >, /, ?]",
"index_uuid": "na",
"index": "filebeat-
"
}
],
"type": "invalid_index_name_exception",
"reason": "Invalid index name [filebeat-
], must not contain the following characters [ , ", , \, <, |, ,, >, /, ?]",
"index_uuid": "na",
"index": "filebeat-
"
},
"status": 400
}


(DharaniKumar) #15

i loaded my filebeat index pattern with timestamp field, so that i dont have to create filebeat index each time(eg. if i start filebeat today it will create filebeat-2017.08.04 index and so on when i start filebeat each day) when filebeat is reading some log files from my machine.


(Magnus Bäck) #16

My point is: If geoip.location is the field you've mapped as geo_point it's not surprising that the location field isn't recognized as geo_point. If you want location to be a geo_point you need to make sure that field is mapped as geo_point.

PUT /filebeat-*/?pretty

you're trying to create an index named, literally, filebeat-*. That's obviously not legal and isn't want you want to do anyway.

{
“mappings”: {
“syslog”: {
“properties”: {
“geoip.location”: {
“type”: “geo_point”
}
}
}
}
}

That's not how the mapping of a nested field is set. Consult the ES documentation for details.


(DharaniKumar) #17

@magnusbaeck as you said it’s not surprising that the location field isn’t recognized as geo_point, i do set location field as geo_point earlier. When i referred in someother disucssion thread someone said that map location as geoip.location and it still not working well as when i had geo_point in location field.


(DharaniKumar) #18

And also when use GET _template/filebeat command, im getting
"filebeat": {
"order": 0,
"template": "filebeat-",
"settings": { .....
..............
.............}}
If index pattern template name is filebeat-
, then index name must be filebeat right?
if so when i used filebeat as index name kibana throws an error like invalid index name .


(Magnus Bäck) #19

i do set location field as geo_point earlier.

What do the actual mappings of the index look like? Use the get mapping API.

If index pattern template name is filebeat-*, then index name must be filebeat right?

It must begin with "filebeat-", yes, but anything can follow after that. It's a wildcard pattern.


(DharaniKumar) #20

I tried get mapping api in 2 different ways.

  1. GET filebeat/_mapping and it throws an error:

{
"error": {
"root_cause": [
{
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "filebeat",
"index_uuid": "na",
"index": "filebeat"
}
],
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "filebeat",
"index_uuid": "na",
"index": "filebeat"
},
"status": 404
}

  1. GET filebeat-*/_mapping and it shows nothing than {}.