Problem
- I am not able to plot any coordinate map as index pattern does not contain any of the following field types: geo_point.
- Upon further investigating the index pattern - the geoip.location has properties [loc, lat] but on request from the elastic search for the mapping the following key is not of type geo_point.
- I am utilising logstash "COMBINEDAPACHELOG" capability with geoip plugin to parse all the keys [ keys such as country_name, city_name are obtained from the same ]
- I could see no geo_point type for geoip.location when i obtained the mapping from elastic search. [Find snippet below]
Current Environment
- Stack being utilised is ELK with filebeat
- Versions
- Elastic Search - 5.5.0v
- Logstash - 5.5.0v
- Kibana - 5.5.0v
- Filebeats - 5.5.1v
Log stash Pipeline
input {
beats {
port => 5000
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
index => "apache-logs"
}
}
GET Mapping from Elastic Search - Short Snipper from the actual [Type Mismatch here]
{
"apache-logs": {
"mappings": {
"log": {
"properties": {
"@timestamp": {
"type": "date"
},
"clientip": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"geoip": {
"properties": {
"city_name": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"ip": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"latitude": {
"type": "float"
},
"location": {
"properties": {
"lat": {
"type": "float"
},
"lon": {
"type": "float"
}
}
},
"longitude": {
"type": "float"
}
}
}
}
}
}
}
}
Solutions tried
- Refreshing the index multiple times.
- The map rendering had worked previously with older versions of elastic search 4.xx, not able to replicate the same.
- PUT mapping_index to set type geo_point explicitly which returned the following error
{
"error": {
"root_cause": [
{
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: mapping type is missing;"
}
],
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: mapping type is missing;"
},
"status": 400
}
- Explicitly set geoip.location.lat, long to float as indicated from Digital Ocean Blog
- Followed the steps exactly from Elastic blog but still did not resolve the issue
I would be grateful if anyone can help me resolve this issue.