Visualize GEO location


(John Immanuel) #1

I am trying to visualize GEO locations using ELK but getting an exception while selecting the bucket GEO coordinates "No Compatible Fields: The "____" index pattern does not contain any of the following field types: geo_point".

Please kindly help me to resolve this issue and get working. Thank in advance.

logstash conf:

input {
stdin { }
}

filter {
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
}
}

date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}

geoip {
source => "clientip"
target => "geoip"
database => "/data/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}

mutate {
	 convert => [ "[geoip][coordinates]", "float"]
}	

useragent {
source => "agent"
target => "useragent"
}
}

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "localhost:9200"
index => "apache_elk_example"
template => "/data/apache_template.json"
template_name => "apache_elk_example"
template_overwrite => true
}

template json:

{
"template": "apache_elk_example",
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"default": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match_mapping_type": "string",
"match": "message"
}
},
{
"string_fields": {
"mapping": {
"index": "analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"index": "not_analyzed",
"ignore_above": 256,
"type": "string"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "float" },
"longitude" : { "type" : "float" }
}
},
"@version": {
"index": "not_analyzed",
"type": "string"
},
"location" : { "type": "geo_point" }
},
"_all": {
"enabled": true
}
}
}
}


(Magnus Bäck) #2

What are the actual mappings of the index? What does an example event look like (copy/paste from Kibana's JSON tab)?


(John Immanuel) #3

Hi Magnus,

Please find below the mapping.

{
"_index": "apache_elk_example",
"_type": "logs",
"_id": "AV4yrDrm8y3sHjU_dCQb",
"_version": 1,
"_score": null,
"_source": {
"request": "/files/grok/?C=N;O=A",
"geoip": {
"timezone": "Europe/Amsterdam",
"ip": "5.10.83.53",
"latitude": 52.35,
"continent_code": "EU",
"city_name": "Amsterdam",
"country_name": "Netherlands",
"country_code2": "NL",
"country_code3": "NL",
"region_name": "North Holland",
"location": {
"lon": 4.9167,
"lat": 52.35
},
"postal_code": "1091",
"region_code": "NH",
"longitude": 4.9167
},
"auth": "-",
"ident": "-",
"verb": "GET",
"message": "5.10.83.53 - - [20/May/2015:21:05:59 +0000] "GET /files/grok/?C=N;O=A HTTP/1.1" 200 3894 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"",
"@timestamp": "2015-05-20T21:05:59.000Z",
"response": "200",
"bytes": "3894",
"clientip": "5.10.83.53",
"@version": "1",
"host": "JOHN2403",
"httpversion": "1.1",
"timestamp": "20/May/2015:21:05:59 +0000"
},
"fields": {
"@timestamp": [
1432155959000
]
},
"sort": [
1432155959000
]
}


(Magnus Bäck) #4

That's an example doc, not the mappings of the index. Use the get mapping API for that. The example document looks fine, but if the index template hasn't been applied things won't work anyway.


(John Immanuel) #5

{
"apache_elk_example":{
"mappings":{
"logs":{
"properties":{
"@timestamp":{
"type":"date"
},
"clientip":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"geoip":{
"properties":{
"city_name":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"continent_code":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"coordinates":{
"type":"float"
},
"country_code2":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"country_code3":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"country_name":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"dma_code":{
"type":"long"
},
"ip":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"latitude":{
"type":"float"
},
"location":{
"properties":{
"lat":{
"type":"float"
},
"lon":{
"type":"float"
}
}
},
"longitude":{
"type":"float"
},
"postal_code":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"region_code":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"region_name":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"timezone":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
}
},
"latitude":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"location":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"longitude":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"message":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
"verb":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
}
}
}
}
}


(Magnus Bäck) #6

Next time please format this kind of stuff as preformatted text so it has a chance of becoming readable.

The [geoip][location] field isn't mapped as geo_point. Perhaps you created/updated the index template after that field had already been mapped. Reindex your data or just drop/recreate the index if you only have test data in it.


(John Immanuel) #7

Thanks for your valuable inputs. I have dropped and recreated new index (apache_elk_example to apache_elk) but getting the same error.


(Magnus Bäck) #8

So the mappings for the index haven't changed? Have you verified that the index template is actually present in ES? I would, without using Logstash,

  • drop the current apache_elk_example index,
  • doublecheck the presence and contents of the apache_elk_example index template,
  • create a new apache_elk_example index and check its mappings.

(John Immanuel) #9

As per your inputs I have tried as below but the [geoip][location] field isn’t mapped as geo_point.

  • Dropped the index and verified using http://localhost:9200/_cat/indices/ (Index apache_elk_example was removed). Recreated new index as "apache_elk"
  • Checked the contents of the template and template name is now updated as "template": "apache_elk".

"location": {
"properties": {
"lat": {
"type": "float"
},
"lon": {
"type": "float"
}
}
}


(Magnus Bäck) #10

Did you update the template before or after you created the index?

You know that the template field in the index template supports wildcards, right?


(John Immanuel) #11

Yes Magnus, I have updated the template as well :disappointed_relieved:


(Magnus Bäck) #12

"Yes" isn't a valid answer to the question "did you update the template before or after you created the index".


(John Immanuel) #13

Its working Magnus. I got the answer from your reply.

Thanks a lot :joy:


(John Immanuel) #14

I am getting another exception while importing the kibana dashboard json. Could you please suggest me a solution?

"Importing Unique Visitors (Unique-Visitors) failed: Could not locate that index-pattern-field (id: geoip.ip.raw)"


(John Immanuel) #15

Now its working.


(system) #16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.