It seemes that when useing the geoip filter. It creates a mapping to convert the location to a geo_point. But only for index's called logstash-*. But I named my index's apache-*. I have been trying to utilize the "template" attribute of the elasticsearch output plugin to do this. but it seems, the only way to get the tile map to work with my custom index name is to create my own template using the API.
my logstash configuration:
input {
# logs provided by rsyslog from other servers
syslog { # apache logs
host => "0.0.0.0"
port => 5544
type => "apache"
}
syslog { # kernel logs
host => "0.0.0.0"
port => 5548
type => mssg
use_labels => true
}
}
filter {
if [type] == "apache" {
grok {
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
if [type] == "mssg" {
grok {
match => { "message" => "%{SYSLOGLINE}" }
}
}
}
output {
if [type] == "apache" {
elasticsearch {
hosts => ["elasticserver"]
index => "apache-%{+YYYY.MM.dd}"
# was trying to use # template => /etc/elasticsearch/apache.json
# but could not figure out what to put in apache.json to get this to work
}
}
if [type] == "mssg" {
elasticsearch {
hosts => ["elasticserver"]
index => "mssg-%{+YYYY.MM.dd}"
}
}
}
Running a tile map on the apache-* index at this point would not work, and kibana would throw an error
saying ''no geo_point found''.
results of GET _template before creating my own:
{
"logstash": {
"order": 0,
"template": "logstash-*",
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"default": {
"dynamic_templates": [
{
"message_field": {
"mapping": {
"fielddata": {
"format": "disabled"
},
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match_mapping_type": "string",
"match": "message"
}
},
{
"string_fields": {
"mapping": {
"fielddata": {
"format": "disabled"
},
"index": "analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"_all": {
"omit_norms": true,
"enabled": true
},
"properties": {
"@timestamp": {
"type": "date"
},
"geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"latitude": {
"type": "float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "float"
}
}
},
"@version": {
"index": "not_analyzed",
"type": "string"
}
}
}
},
"aliases": {
}
}
}
as you can see, the geo_point mapping is there. This was put in by default.
this is how I created my own template PUT /_template/apache-template:
{
"order": 0,
"template": "apache-*",
"settings": {
"index": {
"number_of_shards": "1"
}
},
"mappings": {
"apache": {
"properties": {
"geoip": {
"dynamic": true,
"type": "object",
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
},
"aliases": {}
}
after this the tile map would work. I used this same code I have here for apache-template in the template => /path/to/template.json, but
that did not work.
- is there a way to utilize the template attribute in the elasticsearch output to handle this mapping?
- am I doing this wrong?