Logstash shards with unaassigned replicas

[kostja79]

Hi to all,

I have a cluster with only 1 node.

However, when I start logstash, it creates several replicas of any shard:

GET /_cat/shards?h=index,shard,prirep,state,unassigned.reason 

logstash-2018.04.03               3 p STARTED    
logstash-2018.04.03               3 r UNASSIGNED INDEX_CREATED
logstash-2018.04.03               1 p STARTED    
logstash-2018.04.03               1 r UNASSIGNED INDEX_CREATED
logstash-2018.04.03               2 p STARTED    
logstash-2018.04.03               2 r UNASSIGNED INDEX_CREATED
logstash-2018.04.03               4 p STARTED    
logstash-2018.04.03               4 r UNASSIGNED INDEX_CREATED
logstash-2018.04.03               0 p STARTED    
logstash-2018.04.03               0 r UNASSIGNED INDEX_CREATED

How can I prevent the creation of the replicas when I have only 1 node?

I tried to modify the setting "number_of_replicas" in the template, but probably it has been overriden when logstash started.

Thanks

[Evesy (Michael Eves)]

Hey Kostja,

You can verify if the template has been overwritten by hitting the Elasticsearch API: GET {{url}}:9200/_template/logstash -- This should show how many replicas the Logstash template has configured.

However one thing to note is that the template is only used on the creation of new indices, so if your index logstash-2018.04.03 already exists, changing the template won't have any affect on that index. Instead you can see the settings for that index: GET {{url}}:9200/logstash-2018.04.03/_settings. If you wish to change the number of replicas on an existing index you can do that with the below:

PUT {{url}}:9200/logstash-2018.04.03/_settings

{
  "index": {
	"number_of_replicas": "1"
  }
}

I'm not intimately familiar with the default Logstash templates, but I imagine it won't override anything is a template already exists.

Hope this helps, let me know how you get on

Cheers,
Mike

[kostja79]

Hi Mike,

thanks for your reply.

This is the output of GET _template/logstash*:
{
"logstash-index-template": {
"order": 0,
"index_patterns": [
".logstash"
],
"settings": {
"index": {
"number_of_shards": "1",
"auto_expand_replicas": "0-1",
"codec": "best_compression"
}
},
....

So, basing on the value of auto_expand_replicas, I was expecting that no replicas should be created since I have only 1 node. I do not understand why it creates the replicas.

No problem for the already created indices, I am focused to avoid this issue for the new indices.

Thanks
kostja

[kostja79]

Hi, I solved this issue.

It has been originated by filebeat where there was the following setting:
index.number_of_shards: 3

I changed to index.number_of_shards: 1 and now logstash does not create unallocated shards.

For the current load of my cluster it is fine, I'm going to change it when the load will grow.