On one of the clusters however, the logstash indices are created with only 1 shard, and the
number_of_shards
for the logstash daily indices show as count of 1, even though the mapping loaded in ES has 40.
Any idea what might be causing this ? I can confirm that the logstash nodes are using the correct mapping template and as pointed earlier the template is loaded in Elasticsearch. I don't see any errors in the logs.
You have 2 index template that match logstash-*
The 2 I’m asking you to dump above.
I expect one of them will have number_of_shards : 1 and the other 40
They are both order : 0 templates meaning this is already non sensical because having 2 order 0 template that match an index name at index creation time means you get non-deterministic merging behavior for the multiple template that match and have the same order. You always need to have only 1 template that match or to have many but with different order value so they get applied from lowest to highest in a deterministic way.
At this point I would guess the template named logstash is the default one and because starting with ES7 templates come with 1 shard unless the user overrides it, this template currently sets the number of shards to 1. We’ll see in your dump of both template if one of them sets the shards to 1.
You are indeed correct, the default logstash template has "number_of_shards": "1" and the other one has 40. It's strange because all 15 daily indices on that cluster have shards set to 1 and all of the other 3 clusters and their daily indices have consistently 40, no mix of either. I assume I can just delete the default index with curl -X DELETE "localhost:9200/_template/logstash" ?
Yes you can delete it but it came from logstash feature of auto loading index templates, so you have to be careful because you or someone else has, at least once in the past, configured logstash to load that template or it was done manually.
In short you see my point, make sure you read the logstash ES output doc and understand how logstash is configured to load or NOT load index template by itself into ES or your issue will come back.
About the fact that you get non-deterministic behavior, that part is normal.
What makes this confusing is that all 4 ES clusters have been configured through a configuration management system (Chef in this case) and all the other 3 clusters have identical configs for Elasticsearch and Logstash, including the template. Even listing the templates yields identical result. In any case, thank you for the reply, I'll just delete the default logstash template from all ES clusters.
Hehe to be clear I’m insisting because you insist that something strange or confusing is at play... and it’s not the case:
Multiple matching templates with the same order value will result in a non-deterministic merging order.
Ref:
Which means that the end result config your config management yields is documented as yielding non-deterministic merging behavior. Which is exactly what “doing the same thing 4 times and not getting the same result every time” mean.
I mean its not strange or confusing that you don’t get at least consistent results because the doc specifically says that doing what you’re doing WILL yield inconsistent results.
Maybe I’m bad at getting this point across but nothing weird or surprising is happening.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.