Hey there! I am analysing access logs for a project. I have everything set up, Logstash is doing what is supposed to do, Elasticsearch and Kibana, too (using 5.2). BUT:
Yesterday I decided to write my own template, since I wanted to use the Path Hierarchy Tokenizer. But whenever I index data, my template is just not being used correctly and I don't know why.
The output in my logstash.conf looks like this:
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => "true"
template_overwrite => "false"
}
}
I have tried different things: set a template_name (no effect), set manage_template to "false" (which did nothing), set template_overwrite to "true" (which of course then did overwrite my template with the logstash one), all because I had no idea what to change.
My template looks like this and I used the Dev Tools ind Kibana to write it.
PUT _template/logstash
{
"template": "logstash",
"index_patterns": "logstash-*",
"settings": {
"analysis": {
"analyzer": {
"custom_path_tree": {
"tokenizer": "custom_hierarchy"
},
"custom_path_tree_reversed": {
"tokenizer": "custom_hierarchy_reversed"
}
},
"tokenizer": {
"custom_hierarchy": {
"type": "path_hierarchy",
"delimiter": "/"
},
"custom_hierarchy_reversed": {
"type": "path_hierarchy",
"delimiter": "/",
"reverse": "true"
}
}
}
},
"mappings": {
"logs": {
"properties": {
"object": {
"type": "text",
"fields": {
"tree": {
"type": "text",
"analyzer": "custom_path_tree"
},
"tree_reversed": {
"type": "text",
"analyzer": "custom_path_tree_reversed"
}
}
},
"referral": {
"type": "text",
"fields": {
"tree": {
"type": "text",
"analyzer": "custom_path_tree"
},
"tree_reversed": {
"type": "text",
"analyzer": "custom_path_tree_reversed"
}
}
},
"datetime": {
"type": "date",
"format": "time_no_millis"
},
"size": {
"type": "integer"
}
}
}
}
}
GET _template/logstash results in showing me my template, the way I want it. However, once I index data, nothing happens the way I expect it. The standard logstash template seems to not be used, because once I define an index pattern in Kibana, fields like geoip.latitude don't appear as opposed to not using a custom template. But also the fields like result.tree don't appear and datetime ends up being indexed as a keyword.
http://localhost:9200/_all/_mapping?pretty=1 yields this:
{
"logstash-2019.07.03" : {
"mappings" : {
"logs" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"action" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"datetime" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
} ...
I am probably missing something very simple, but this is the first time I am working with the ELK stack and I am only asking because I don't know what to do anymore. I would be grateful if someone had an idea.
