Visualize GEO location

Elastic Stack Logstash
1

I am trying to visualize GEO locations using ELK but getting an exception while selecting the bucket GEO coordinates "No Compatible Fields: The "____" index pattern does not contain any of the following field types: geo_point".

Please kindly help me to resolve this issue and get working. Thank in advance.

logstash conf:

input {
stdin { }
}

filter {
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
}
}

date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}

geoip {
source => "clientip"
target => "geoip"
database => "/data/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}

mutate {
	 convert => [ "[geoip][coordinates]", "float"]
}

useragent {
source => "agent"
target => "useragent"
}
}

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "localhost:9200"
index => "apache_elk_example"
template => "/data/apache_template.json"
template_name => "apache_elk_example"
template_overwrite => true
}

template json:

{
"template": "apache_elk_example",
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"default": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match_mapping_type": "string",
"match": "message"
}
},
{
"string_fields": {
"mapping": {
"index": "analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"index": "not_analyzed",
"ignore_above": 256,
"type": "string"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "float" },
"longitude" : { "type" : "float" }
}
},
"@version": {
"index": "not_analyzed",
"type": "string"
},
"location" : { "type": "geo_point" }
},
"_all": {
"enabled": true
}
}
}
}

2

What are the actual mappings of the index? What does an example event look like (copy/paste from Kibana's JSON tab)?

3

Hi Magnus,

Please find below the mapping.

{
"_index": "apache_elk_example",
"_type": "logs",
"_id": "AV4yrDrm8y3sHjU_dCQb",
"_version": 1,
"_score": null,
"_source": {
"request": "/files/grok/?C=N;O=A",
"geoip": {
"timezone": "Europe/Amsterdam",
"ip": "5.10.83.53",
"latitude": 52.35,
"continent_code": "EU",
"city_name": "Amsterdam",
"country_name": "Netherlands",
"country_code2": "NL",
"country_code3": "NL",
"region_name": "North Holland",
"location": {
"lon": 4.9167,
"lat": 52.35
},
"postal_code": "1091",
"region_code": "NH",
"longitude": 4.9167
},
"auth": "-",
"ident": "-",
"verb": "GET",
"message": "5.10.83.53 - - [20/May/2015:21:05:59 +0000] "GET /files/grok/?C=N;O=A HTTP/1.1" 200 3894 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"",
"@timestamp": "2015-05-20T21:05:59.000Z",
"response": "200",
"bytes": "3894",
"clientip": "5.10.83.53",
"@version": "1",
"host": "JOHN2403",
"httpversion": "1.1",
"timestamp": "20/May/2015:21:05:59 +0000"
},
"fields": {
"@timestamp": [
1432155959000
]
},
"sort": [
1432155959000
]
}

4

That's an example doc, not the mappings of the index. Use the get mapping API for that. The example document looks fine, but if the index template hasn't been applied things won't work anyway.

5

{
"apache_elk_example":{
"mappings":{
"logs":{
"properties":{
"@timestamp":{
"type":"date"
},
"clientip":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"geoip":{
"properties":{
"city_name":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"continent_code":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"coordinates":{
"type":"float"
},
"country_code2":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"country_code3":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"country_name":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"dma_code":{
"type":"long"
},
"ip":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"latitude":{
"type":"float"
},
"location":{
"properties":{
"lat":{
"type":"float"
},
"lon":{
"type":"float"
}
}
},
"longitude":{
"type":"float"
},
"postal_code":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"region_code":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"region_name":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"timezone":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
}
},
"latitude":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"location":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"longitude":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"message":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
"verb":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
}
}
}
}
}

6

Next time please format this kind of stuff as preformatted text so it has a chance of becoming readable.

The [geoip][location] field isn't mapped as geo_point. Perhaps you created/updated the index template after that field had already been mapped. Reindex your data or just drop/recreate the index if you only have test data in it.

7

Thanks for your valuable inputs. I have dropped and recreated new index (apache_elk_example to apache_elk) but getting the same error.

8

So the mappings for the index haven't changed? Have you verified that the index template is actually present in ES? I would, without using Logstash,

  • drop the current apache_elk_example index,
  • doublecheck the presence and contents of the apache_elk_example index template,
  • create a new apache_elk_example index and check its mappings.
9

As per your inputs I have tried as below but the [geoip][location] field isn’t mapped as geo_point.

  • Dropped the index and verified using http://localhost:9200/_cat/indices/ (Index apache_elk_example was removed). Recreated new index as "apache_elk"
  • Checked the contents of the template and template name is now updated as "template": "apache_elk".

"location": {
"properties": {
"lat": {
"type": "float"
},
"lon": {
"type": "float"
}
}
}

10

Did you update the template before or after you created the index?

You know that the template field in the index template supports wildcards, right?

11

Yes Magnus, I have updated the template as well :disappointed_relieved:

12

"Yes" isn't a valid answer to the question "did you update the template before or after you created the index".

13

Its working Magnus. I got the answer from your reply.

Thanks a lot :joy:

14

I am getting another exception while importing the kibana dashboard json. Could you please suggest me a solution?

"Importing Unique Visitors (Unique-Visitors) failed: Could not locate that index-pattern-field (id: geoip.ip.raw)"

15

Now its working.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.