Index template not taking effect on index created

Elastic Stack Logstash
1

Hello guys,

i have a problem on my index creation, the i can see that mappings/settings is created based on the template as checked (GET _template). but the created index doesnt get the mappings/settings when created.

Here is my config for the output

output {
if [type] == "log" {
elasticsearch {
hosts => ["awsES:443"]
sniffing => false
index => "em-hlr-applog-%{+YYYY.MM.dd}"
template => "/home/ubuntu/ELK/my-emhlrlog-template-updated.json"
template_name => "log-"
template_overwrite => true
document_type => "%{[@metadata][type]}"
}
}
else if [type] == "sys" {
elasticsearch {
hosts => ["awsES:443"]
sniffing => false
manage_template => false
index => "emhlr-syslog-%{+YYYY.MM.dd}"
template => "/home/ubuntu/ELK/my-syslogtemplate-updated.json"
template_name => "syslog-"
template_overwrite => true
document_type => "%{[@metadata][type]}"
}
}

2

And what does the template look like?

3

Hello Magnus,

Please see below. Index template already take effect. But is it possible to have multiple mappings for different "type" (document_type in filebeat) on a single template file.

like ex. i have the following "type".

webapplog
feapplog
uiapplog
hlrlog

----template---

{

"template": "*applog-*",

"settings": {

    "analysis": {

        "index" : {



            "number_of_shards" : 3,


            "number_of_replicas" : 1



        },

        "analyzer": {

            "domain_name_analyzer": {

                "filter":"lowercase",

                "tokenizer": "domain_name_tokenizer",

                "type": "custom"

            }

        },

        "tokenizer": {

            "domain_name_tokenizer": {

                "type": "PathHierarchy",

                "delimiter": ".",

                "reverse": true

            }

        }

    }

},

"mappings": {

"hlrlog": {

  "properties": {

      "message": {

        "type": "text",

        "fields": {

          "ws": {

            "type":     "text",

            "analyzer": "whitespace"

          }

        }

      }

    }

  }

}

}

4

So what happens if you create an index named e.g. em-hlr-applog-2017.12.31 with a REST call to ES? What mappings does the index get?

Please see below. Index template already take effect. But is it possible to have multiple mappings for different "type" (document_type in filebeat) on a single template file.

Yes.

5

here magnus, you can see i have multiple mappings that should not be included. I've used multiple index template file for each "type".

{
"em-hlr-applog-2017.12.07" : {
"aliases" : { },
"mappings" : {
"maprouterlog" : {
"properties" : {
"message" : {
"type" : "text",
"fields" : {
"ws" : {
"type" : "text",
"analyzer" : "whitespace"
}
}
}
}
},
"hlrlog" : {
"properties" : {
"%{" : {
"properties" : {
"@metadata" : {
"properties" : {
"fingerprint" : {
"properties" : {
"}" : {
"type" : "long"
}
}
}
}
}
}
},
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"akkatimestamp" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"beat" : {
"properties" : {
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"dateko" : {
"type" : "date"
},
"host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"input_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"message" : {
"type" : "text",
"fields" : {
"ws" : {
"type" : "text",
"analyzer" : "whitespace"
}
}
},
"offset" : {
"type" : "long"
},
"received_at" : {
"type" : "date"
},
"received_from" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"source" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tags" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"timestampko" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"applog" : {
"properties" : {
"message" : {
"type" : "text",
"fields" : {
"ws" : {
"type" : "text",
"analyzer" : "whitespace"
}
}
}
}
},
"emggsnlog" : {
"properties" : {
"message" : {
"type" : "text",
"fields" : {
"ws" : {
"type" : "text",
"analyzer" : "whitespace"
}
}
}
}
}
},
"settings" : {
"index" : {
"number_of_shards" : "5",
"provided_name" : "em-hlr-applog-2017.12.07",
"creation_date" : "1512604806737",
"analysis" : {
"index" : {
"number_of_shards" : "3",
"number_of_replicas" : "1"
},
"analyzer" : {
"domain_name_analyzer" : {
"filter" : "lowercase",
"type" : "custom",
"tokenizer" : "domain_name_tokenizer"
}
},
"tokenizer" : {
"domain_name_tokenizer" : {
"reverse" : "true",
"type" : "PathHierarchy",
"delimiter" : "."
}
}
},
"number_of_replicas" : "1",
"uuid" : "Ace2Job1RgCc7l8KA8O4Qw",
"version" : {
"created" : "5050299"
}
}
}
}
}

6

here magnus, you can see i have multiple mappings that should not be included.

But that's not the mappings of an index you've created with a REST call is it? I want to see what it looks like before you start adding documents with Logstash.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.