How to change number_of_shards and number_of_replica in elasticseatch template

[Debashis_Adak]

Hi All,

ELK Version: 6.4.0
System Details: Windows
Host: Localhost

I am trying to create a new index in Elasticsearch via Logstash where I need to change the default index settings. By default 5 shards and 1 replica gets created but I want to change the default setting.

I have read some of the blogs and implemented the below in elasticsearch:

PUT _template/logstash-index-template
{
"index_patterns": ["sample_index*"],
"settings": {
"index.refresh_interval": "10s",
"number_of_shards": "1",
"number_of_replicas": "0"
}
}

But when I am creating any new index with the same index pattern it is going for the default settings.

yellow open sample_index-2019.02 A1FbzwvaT7uOSmeGf4wSDw 5 1 10 0 120.1kb 120.1kb

My output filter of Logstash is..

output {
elasticsearch {
hosts => localhost
index => "sample_index-%{+YYYY.MM}"
}
}

Not sure what is needed to be done to change the default settings.

Thanks in advanced...

Regards,
Debashis Adak

[Bernt_Rostad]

@Debashis_Adak

Not sure what is needed to be done to change the default settings.

You could have a default order 0 template overriding it since you didn't specify the order. Index templates are always applied in increasing order, with order 0 first, then order 1, order 2 etc.

Could you try to add it as an order 1 template like this:

PUT _template/logstash-index-template
{
   "index_patterns" : ["sample_index*"],
   "order" : 1
   "settings" : {
       "index.refresh_interval" : "10s",
       "number_of_shards" : "1",
       "number_of_replicas" : "0"
   }
}

[Debashis_Adak]

Hi Bernt,

Thanks for your valuable input. The problem is resolved now.

But I faced one problem. I tried to update the template with with your parameter. But it was not getting updated. It was holding the previous template only.

I tried to delete the existing template by "DELETE /_template/logstash-index-template" and acknowledgement was coming TRUE but template was not getting deleted.

Post delete command whenever I was running "GET /_template/logstash-test-template" it was giving me the result. So, the summary is though I was trying to delete the existing template it was not getting deleted. Not even it was getting updated.

Have you faced similar problem? Also, I will be really thankful if you can explain the ordering a little. I am not sure what is the order of the default template of any index.

[Bernt_Rostad]

@Debashis_Adak

I tried to update the template with with your parameter. But it was not getting updated. It was holding the previous template only.

I'm sorry, I should have tested the example before I posted it. I see now that it fails with parser error and there are two things that fail: Index template names can't contain dashes ("-") and I forgot a comma after the order field.

This should work better:

PUT _template/logstash_index_template
{
   "index_patterns" : ["sample_index*"],
   "order" : 1,
   "settings" : {
       "index.refresh_interval" : "10s",
       "number_of_shards" : "1",
       "number_of_replicas" : "0"
   }
}

@Debashis_Adak

Have you faced similar problem

No, I can overwrite the old index template with a new one, as long as I provide the same valid name

[Bernt_Rostad]

@Debashis_Adak

if you can explain the ordering a little

The order field is very useful because it allows you to build an index template hierarchy, where the lowest order template has the most general settings and the higher orders more specialized.

Let us say you have three index templates like these in your cluster:

PUT _template/general_template
{
   "index_patterns" : ["*"],
   "order" : 0,
   "settings" : {
       "number_of_shards" : "2",
       "number_of_replicas" : "1"
   }
}

PUT _template/myorg_template
{
   "index_patterns" : ["myorg*"],
   "order" : 1,
   "settings" : {
       "number_of_shards" : "3",
   }
}

PUT _template/myorg_weekly_template
{
   "index_patterns" : ["myorg_weekly*"],
   "order" : 2,
   "settings" : {
       "number_of_shards" : "1",
       "number_of_replicas" : "0",
   }
}

Any index name you specify will match the "*" index_pattern of the order 0 template, which means that with this setup all indices in your cluster will by default be created with 2 primary and 1 replica shard.

But, if you create an index with a name starting with "myorg", then the order 1 template kicks in because the name matches its index_pattern. In that case the order 1 template overrides the "number_of_shards" so that new myorg-indices will be created with 3 primary shards (but keeping the "number_of_replicas").

Finally, if you create an index named for instance "myorg_weekly_2019_06" then that name will match the index_pattern of your order 2 template, overriding both "number_of_shards" and "number_of_replicas" setting them to 1 and 0 respectively for that index.

You can go on and add more specialized templates in this manner, the main point is to use more and more precise index_patterns for the higher order templates.

[Debashis_Adak]

Wow!!!! The example was simply awesome. Thanks for the valuable input. Now everything is in order..