Shard count does not match active mapping

I have 4 Elasticsearch 7.1 clusters using the same mapping template, that I can verify being in use:

localhost:9200/_template/mytemplate

...
   "settings": {
  "index": {
     ...
    "number_of_shards": "40",
    "refresh_interval": "5s"
  }
...

On one of the clusters however, the logstash indices are created with only 1 shard, and the

number_of_shards

for the logstash daily indices show as count of 1, even though the mapping loaded in ES has 40.

Any idea what might be causing this ? I can confirm that the logstash nodes are using the correct mapping template and as pointed earlier the template is loaded in Elasticsearch. I don't see any errors in the logs.

What's the output of:

GET /_cat/templates?v&s=name

What's the name(s) of your logstash daily indices?

The output is as follows:

curl -s -X GET 'localhost:9200/_cat/templates?v&s=name'
name                 index_patterns             order      version
.kibana_task_manager [.kibana_task_manager]     0          7010099
.logstash-management [.logstash]                0
.management-beats    [.management-beats]        0          70000
.ml-anomalies-       [.ml-anomalies-*]          0          7010199
.ml-config           [.ml-config]               0          7010199
.ml-meta             [.ml-meta]                 0          7010199
.ml-notifications    [.ml-notifications]        0          7010199
.ml-state            [.ml-state*]               0          7010199
.monitoring-alerts-7 [.monitoring-alerts-7]     0          7000199
.monitoring-beats    [.monitoring-beats-7-*]    0          7000199
.monitoring-es       [.monitoring-es-7-*]       0          7000199
.monitoring-kibana   [.monitoring-kibana-7-*]   0          7000199
.monitoring-logstash [.monitoring-logstash-7-*] 0          7000199
.triggered_watches   [.triggered_watches*]      2147483647
.watch-history-9     [.watcher-history-9*]      2147483647
.watches             [.watches*]                2147483647
logstash             [logstash-*]               0          60001
mytemplate           [logstash-*]               0

The daily index matches the logstash-* pattern, e.g logstash-2019.06.24

I get pretty much identical output on the other 3 ES clusters.

1 Like

Thanks now the output of:

GET /_template/logstash

and the output of:

GET /_template/mytemplate

You have 2 index template that match logstash-*
The 2 I’m asking you to dump above.
I expect one of them will have number_of_shards : 1 and the other 40

They are both order : 0 templates meaning this is already non sensical because having 2 order 0 template that match an index name at index creation time means you get non-deterministic merging behavior for the multiple template that match and have the same order. You always need to have only 1 template that match or to have many but with different order value so they get applied from lowest to highest in a deterministic way.

https://www.elastic.co/guide/en/elasticsearch/reference/7.1/indices-templates.html

At this point I would guess the template named logstash is the default one and because starting with ES7 templates come with 1 shard unless the user overrides it, this template currently sets the number of shards to 1. We’ll see in your dump of both template if one of them sets the shards to 1.

You are indeed correct, the default logstash template has "number_of_shards": "1" and the other one has 40. It's strange because all 15 daily indices on that cluster have shards set to 1 and all of the other 3 clusters and their daily indices have consistently 40, no mix of either. I assume I can just delete the default index with curl -X DELETE "localhost:9200/_template/logstash" ?

Yes you can delete it but it came from logstash feature of auto loading index templates, so you have to be careful because you or someone else has, at least once in the past, configured logstash to load that template or it was done manually.
In short you see my point, make sure you read the logstash ES output doc and understand how logstash is configured to load or NOT load index template by itself into ES or your issue will come back.

About the fact that you get non-deterministic behavior, that part is normal.

What makes this confusing is that all 4 ES clusters have been configured through a configuration management system (Chef in this case) and all the other 3 clusters have identical configs for Elasticsearch and Logstash, including the template. Even listing the templates yields identical result. In any case, thank you for the reply, I'll just delete the default logstash template from all ES clusters.

Hehe to be clear I’m insisting because you insist that something strange or confusing is at play... and it’s not the case:

Multiple matching templates with the same order value will result in a non-deterministic merging order.

Ref:

Which means that the end result config your config management yields is documented as yielding non-deterministic merging behavior. Which is exactly what “doing the same thing 4 times and not getting the same result every time” mean.

I mean its not strange or confusing that you don’t get at least consistent results because the doc specifically says that doing what you’re doing WILL yield inconsistent results.

Maybe I’m bad at getting this point across :wink: but nothing weird or surprising is happening.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.