How to use geoip logstash filter?

Elastic Stack Logstash
1

In my log file i've lat and lon values in separate fields. I cant able to use tile map in kibana.

Kibana.png929×161 13.9 KB

I have these details in my log file. Can some one guide me how to use it in to tilemap in kibana.

Thanks in advance.

2

In short you have to

  • make sure the mapping of your index configures a designated field as a geo_point field (this is typically done via an index template), and
  • store your lat and lon values in that field in one of the formats that ES will recognize as geo_point (this is described in the ES documentation for geo_point).
3

"geoip": {
"dynamic": "true",
"properties": {
"location": {
"type": "geo_point"
}

this is my index template @magnusbaeck.

4

What do the actual mappings of your index look like? What does an example event look like (copy/paste from Kibana's JSON tab in the Discover panel, no screenshots please)?

5

Do we need to download any database and place it in a specified path or kibana itself locate the lat and longi fields?

6

Kibana will locate the fields by itself. Just make sure the index mappings are correct.

7

yeah @magnusbaeck. What about the database?
Do i need to download it from my side or its inbuilt in kibana(5.5.0).
I'm using logstash 5.5.0 too.

8

What database are you talking about?

9

GeoLite2 database?!?!?! A city database to locate geolocation.

10

As documented the geoip filter ships with a GeoIP database so you don't have to do anything about it, although the bundled database might be out of date. The documentation contains a paragraph or two about it.

11

"passcodeEnabled": "false",
"currentTimestamp": "2017-08-04 15:52:00.449",
"name": "Micromax-CBE",
"location": {
"latitude": "11.0130009",
"longitude": "76.9829831"
},

These are some datas that i inserted in location field. Lat and long fields as string.

12

but kibana does not recognise it as geopoints internally.

13

According to your index template geoip.location is the name of your geo_point field.

14

Yeah @magnusbaeck i did set location to geoip.location like as follows:
PUT /filebeat-*/?pretty
{
"mappings": {
"syslog": {
"properties": {
"geoip.location": {
"type": "geo_point"
}
}
}
}
}

and i got this error:
{
"error": {
"root_cause": [
{
"type": "invalid_index_name_exception",
"reason": "Invalid index name [filebeat-], must not contain the following characters [ , ", , \, <, |, ,, >, /, ?]",
"index_uuid": "na",
"index": "filebeat-"
}
],
"type": "invalid_index_name_exception",
"reason": "Invalid index name [filebeat-], must not contain the following characters [ , ", , \, <, |, ,, >, /, ?]",
"index_uuid": "na",
"index": "filebeat-"
},
"status": 400
}

15

i loaded my filebeat index pattern with timestamp field, so that i dont have to create filebeat index each time(eg. if i start filebeat today it will create filebeat-2017.08.04 index and so on when i start filebeat each day) when filebeat is reading some log files from my machine.

16

My point is: If geoip.location is the field you've mapped as geo_point it's not surprising that the location field isn't recognized as geo_point. If you want location to be a geo_point you need to make sure that field is mapped as geo_point.

PUT /filebeat-*/?pretty

you're trying to create an index named, literally, filebeat-*. That's obviously not legal and isn't want you want to do anyway.

{
“mappings”: {
“syslog”: {
“properties”: {
“geoip.location”: {
“type”: “geo_point”
}
}
}
}
}

That's not how the mapping of a nested field is set. Consult the ES documentation for details.

17

@magnusbaeck as you said it’s not surprising that the location field isn’t recognized as geo_point, i do set location field as geo_point earlier. When i referred in someother disucssion thread someone said that map location as geoip.location and it still not working well as when i had geo_point in location field.

18

And also when use GET _template/filebeat command, im getting
"filebeat": {
"order": 0,
"template": "filebeat-",
"settings": { .....
..............
.............}}
If index pattern template name is filebeat-, then index name must be filebeat right?
if so when i used filebeat as index name kibana throws an error like invalid index name .

19

i do set location field as geo_point earlier.

What do the actual mappings of the index look like? Use the get mapping API.

If index pattern template name is filebeat-*, then index name must be filebeat right?

It must begin with "filebeat-", yes, but anything can follow after that. It's a wildcard pattern.

20

I tried get mapping api in 2 different ways.

  1. GET filebeat/_mapping and it throws an error:

{
"error": {
"root_cause": [
{
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "filebeat",
"index_uuid": "na",
"index": "filebeat"
}
],
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "filebeat",
"index_uuid": "na",
"index": "filebeat"
},
"status": 404
}

  1. GET filebeat-*/_mapping and it shows nothing than {}.