Dynamic Template not applying

Elastic Stack Elasticsearch
1

Hi Guys,

I have an issue with my index template not working as I would expect.

I have an index being created from my Filebeats input. This index is called "testlogs" and a type of "testlog".

As I have set a specific index name I had tried to import the filebeats template and change the index and type names to match my specific settings. The issue is that the index does not appear to be being applied.

My newly created (by Logstash GROK filter) fields are being set to be analysed. The Dynamic_template option should set all newly discovered fields to be not_analyzed. My template is below.

"template_2": {
"order": 0,
"template": "testlogs-",
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"testlog": {
"dynamic_templates": [
{
"fields": {
"path_match": "fields.",
"mapping": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"match_mapping_type": "string"
}
}
],
"_all": {
"norms": {
"enabled": false
}
},
"properties": {
"@timestamp": {
"type": "date"
},
"offset": {
"type": "long"
},
"beat": {
"properties": {
"hostname": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"name": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
}
}
},
"input_type": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"source": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"message": {
"norms": {
"enabled": false
},
"index": "analyzed",
"type": "string"
},
"type": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
}
}
}
},
"aliases": {}
}

Is there something wrong with the dynamic_template section is because I have set the index and type name? I understand I could manually map the fields but I would rather have the dynamic mapping set all fields as not_analyzed and then enable analysing on specific fields.

It worked fine before I manually changed the index name from filebeat-* to an actual name.

Any help would be appreciated.

James

2

Your type looks ok, but;

These don't match? What does your LS config look like?

3

Sorry the index is called "testlogs-YYYY.MM.dd" so the template should match on "testlogs-*"

My LS Config is below

input {
beats {
port => 5044
codec => multiline {
pattern => "^[0-2][0-9]-[0-2][0-9]-[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9].[0-9]{3}"
negate => "true"
what => "previous"
max_lines => 600
}
}
}

filter {
if [type] == "OdinBA" {
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{ODINTIME:time} %{NOTSPACE:Process} %{GREEDYDATA:ID} %{PRIORITY:Priority} %{GREEDYDATA:Message}" }
match => { "source" => "%{GREEDYDATA}\%{GREEDYDATA}\%{GREEDYDATA}\%{GREEDYDATA:Filename}.log" }
break_on_match => false
}
date {
match => [ "time", "YY-MM-dd HH:mm:ss.SSS" ]
}
mutate {
gsub => [ "Priority", "]", " " ]
}
}
if [type] == "testlogs" {
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{ODINTIME:time} %{NOTSPACE:Process} %{GREEDYDATA:ID} %{PRIORITY:Priority} %{GREEDYDATA:Message}" }
match => { "source" => "%{GREEDYDATA}\%{GREEDYDATA}\%{GREEDYDATA:Filename}.log" }
break_on_match => false
}
date {
match => [ "time", "YY-MM-dd HH:mm:ss.SSS" ]
}
mutate {
gsub => [ "Priority", "]", " "]
}
}
}

output {
if [type] == "OdinBA" {
elasticsearch {
hosts => ["192.169.38.123:9200","192.168.38.124:9200"]
manage_template => false
index => "odinba-%{+YYYY.MM.dd}"
document_type => "BALog"
}
}
if [type] == "testlogs" {
elasticsearch {
hosts => ["192.168.38.123:9200","192.168.38.124:9200"]
manage_template => false
index => "testlogs-%{+YYYY.MM.dd}"
document_type => "testlog"
}
}
}

My config splits the filebeat input into 2 different indexes and sets the type. This seems to work fine and both indexes appear. Elasticsearch sees the index and shows the document_type as testlog so the output looks okay.