Elasticsearch isn't making new indices have replicas

Elastic Stack Elasticsearch
1

Despite:

index.number_of_replicas: 2

in /etc/elasticsearch/elasticsearch.yml on my cluster nodes

new indices are not being replicated:

summary | awk '$5 == 0'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-2016.02.20 5 0 3536958 0 1gb 1gb
green open logstash-2016.02.21 5 0 3656177 0 1.1gb 1.1gb
green open logstash-2016.02.22 5 0 5557430 0 1.5gb 1.5gb
green open logstash-2016.02.23 5 0 4782864 0 1.1gb 1.1gb
green open logstash-2016.02.24 5 0 3079991 0 853.9mb 853.9mb

Past ones after a manual PUT of replica count as 2 are all 2 up to the 20th.

The elasticsearch cluster nodes has been rebooted several times since the

index.number_of_replicas: 2

was put into elasticsearch.yml weeks ago.

My question is why isn't elasticsearch automatically setting up replicas for the new daily indices?

Thanks for any thoughts.

2

Hard to say from just this as to what could be causing the issue. In the past when I've come across the same problem its been because I had tagged the index to go to a hot node, and there wasn't enough space any hot nodes to store the data. Have you set node-type on any of your nodes?

3

It's because the template for Logstash defines a single replica, and templates take precedence over that setting.

4

Do you know where the template is defined?

I can of course run a script to replicate logstash indexes automatically via cron daily but would like to take the other road if it is even feasible to change the logstash template.

5

Check the _template API endpoint.

6

Okay, I've put this in on all the ES cluster nodes:

cat /etc/elasticsearch/templates/logstash.template
{
"logstash" : {
"order" : 0,
"template" : "logstash-",
"settings" : {
"index.refresh_interval" : "5s",
"index.number_of_replicas": "2",
"index.number_of_shards:" "5"
},
"mappings" : {
"default" : {
"dynamic_templates" : [ {
"string_fields" : {
"mapping" : {
"index" : "analyzed",
"omit_norms" : true,
"type" : "string",
"fields" : {
"raw" : {
"index" : "not_analyzed",
"ignore_above" : 256,
"type" : "string"
}
}
},
"match_mapping_type" : "string",
"match" : ""
}
} ],
"properties" : {
"geoip" : {
"dynamic" : true,
"path" : "full",
"properties" : {
"location" : {
"type" : "geo_point"
}
},
"type" : "object"
},
"@version" : {
"index" : "not_analyzed",
"type" : "string"
}
},
"_all" : {
"enabled" : true
}
}
},
"aliases" : { }
}
}

7

Be mindful that file based templates went away in Elasticsearch 2.0.

8

Also, two replicas is overkill.