Template not taking effect for ES output

Elastic Stack Logstash
1

Hello,

I am trying to change the type of a field, so i am using a template option in my elastic search output plugin.

My output field is like;

output
{
	#stdout { codec => rubydebug }
	if "uat-" in [app]
	{
		elasticsearch
		{
			index => "geo"
			hosts => ["localhost:9200"]
			template => "D:\ELK\server\elk-conf-template.json"
			template_name => "elk-conf-template"
			template_overwrite => true
		}
	}
}

and the template is like;

{
  "template": "elk-conf-template",
  "settings": {
     "index.refresh_interval": "5s"
  },

  "mappings": {
    "doc": {
      "properties": {
        "geoip": {
          "properties": {
            "location": {
			  "type" : "geo_point"
            }
          }
        },
        "ip": {
          "type": "ip",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

When i try to, PUT geo the same from Dev Tools in Kibana, it properly maps the location type to geo_point but if i am trying to do the same via a template, it does not take effect. It maps the location to a text type by default.

Could anybody please help?

Regards

Kaushik

2

What determines which index template is applied is the index_patterns (ES 6.x and later) or template (ES 5.x and earlier) setting in the template. Your template setting says "elk-conf-template" so it'll only match newly created indexes with that exact name. See the ES documentation for details.

3

Hey @magnusbaeck,

I do not quite follow you. By index_pattern do you mean its a setting in elasticsearch?

I did go through the documentation, and could not find it.

Are you saying the template name and the index name must match for it to take effect?

Regards

Kaushik Vankayala

4

It's a setting in the index template JSON blob that controls whether the template applies to a new index. In 5.x that setting is called template and in 6.x it's called index_patterns. See the documentation for your version of ES for details.

5

Could you please give an example as to how to work with the template setting of the elasticsearch output plugin?

I am unable to infer its use!
FYI: I am using a enterprise cloud instance for E and K and the L is running on my local machine with version 6.3!

So, my understanding is to set the type of some specific fields rather than the default (which is string type) we use template setting in elasticsearch output plugin which will tell elastic to store those fields as indicated in the template file.

As explained above when i PUT the mapping in dev tools of kibana before ingesting the data i can get the fields as expected.

But i would like to do that with the help of logstash conf file only - the data type setting!

Please kindly help or correct me if i am wrong!

6

Could you please give an example as to how to work with the template setting of the elasticsearch output plugin?

The template option in the elasticsearch output only selects which template file is uploaded to Elasticsearch. Nothing else. It does not influence which template is applied to the index you're writing to.

As I said, the deciding factor of which template or templates apply to a newly created index is the template'/index_patterns` setting in the template itself.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.