ES not creating indexes based on template

[jlixfeld (Jason)]

Hi there!

I have a daily index that is created from an index template, but I made changes to the index template (increased the number of shards from 3 to 4) however the new indexes are still created with only 3 shards. Can't seem to figure out why.

I tried some pretty heavy-handed approaches to try and make it work. The last of which was to shut down Logstash, delete the index, delete the index template, then restart Logstash. But to no avail.

root@ls1:/home/jlixfeld# systemctl stop logstash
root@ls1:/home/jlixfeld#

root@eskb:/var/log/elasticsearch# curl -s -X DELETE http://localhost:9200/elastiflow-3.3.0-2018.10.17
{"acknowledged":true}
root@eskb:/var/log/elasticsearch# curl -s -X DELETE http://localhost:9200/_template/elastiflow-3.3.0
{"acknowledged":true}
root@eskb:/var/log/elasticsearch# curl -s -X GET http://localhost:9200/_template/elastiflow-3.3.0
{}
root@eskb:/var/log/elasticsearch# curl -s -X GET http://localhost:9200/elastiflow-3.3.0-2018.10.17
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"elastiflow-3.3.0-2018.10.17","index":"elastiflow-3.3.0-2018.10.17"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"elastiflow-3.3.0-2018.10.17","index":"elastiflow-3.3.0-2018.10.17"},"status":404}root@eskb:/var/log/elasticsearch#

root@ls1:/home/jlixfeld# systemctl start logstash
root@ls1:/home/jlixfeld#

root@eskb:/var/log/elasticsearch# curl -s -X GET http://localhost:9200/elastiflow-3.3.0-2018.10.17
...
"number_of_shards":"3"
...
root@eskb:/var/log/elasticsearch# curl -s -X GET http://localhost:9200/_template/elastiflow-3.3.0
...
"number_of_shards":"4"
...

Running ELK 6.4.2

What am I missing?

[dadoonet (David Pilato)]

Can't see all your templates here so hard to know...

[jlixfeld (Jason)]

Hi, sorry I didn't think it was appropriate to dump all 12,000+ lines of the template here, so I assumed illustrating simply that I had changed the number of shards was acceptable info.

The original template is located here:

robcowart/elastiflow/blob/master/logstash/elastiflow/templates/elastiflow.template.json

All that has changed is number_of_shards. And replicas, actually, I had forgotten about that.

Anyway, here's a diff from the original template above and the version I modified:

root@ls1:/home/jlixfeld# wget https://raw.githubusercontent.com/robcowart/elastiflow/master/logstash/elastiflow/templates/elastiflow.template.json
--2018-10-17 10:24:33--  https://raw.githubusercontent.com/robcowart/elastiflow/master/logstash/elastiflow/templates/elastiflow.template.json
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.124.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.124.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 315636 (308K) [text/plain]
Saving to: ‘elastiflow.template.json.2’

elastiflow.template.json.2               100%[=================================================================================>] 308.24K  --.-KB/s    in 0.008s

2018-10-17 10:24:33 (37.6 MB/s) - ‘elastiflow.template.json.2’ saved [315636/315636]

root@ls1:/home/jlixfeld# diff elastiflow.template.json.2 /etc/logstash/elastiflow/templates/elastiflow.template.json
3c3
<   "version": 30300,
---
>   "version": 30301,
7,8c7,8
<       "number_of_shards": 3,
<       "number_of_replicas": 1,
---
>       "number_of_shards": 4,
>       "number_of_replicas": 0,
root@ls1:/home/jlixfeld#

And an excerpt of the startup log referencing that modified template:

[2018-10-17T09:38:01,240][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elastiflow/templates/elastiflow.template.json"}

If you'd still like to see the full output of the modified version, perhaps I can stick it on pastebin or something?

[Prashant_Agrawal]

My gut feeling says that the order of your current template is 0 , which means it would have been applied first and later on you have some template which also matches your index pattern and has some different config and that could be overwriting the earlier config with new one,

Just check if you have any template which also matches this index pattern , may be a template with * as index pattern ?

[jlixfeld (Jason)]

Your gut was right! There was another template that had the same index_patterns and with the old settings. Deleting that old template and deleting today's index created a new index with 4 shards.

Awesome, thank you!

{
  "elastiflow-3.1.0": {
    "order": 0,
    "version": 30300,
    "index_patterns": [
      "elastiflow-3.3.0-*"
    ],
    "settings": {
      "index": {
        "codec": "best_compression",
        "refresh_interval": "10s",
        "number_of_shards": "3",
        "number_of_replicas": "0"
      }
    },

[Prashant_Agrawal]

Great..