No number-of-replicas/shards ... what then?

[alexolivan (Alejandro Olivan)]

Hi again community!

I have just created a small cluster of VMs to keep with my experiments with new ELK5 and filebeat, and, to my surprise, I was unable to start EL until I removed shards/replica config from yml file...

By reading around, I readed that, those values have to be manually set per index.... and of course I can't believe this.
There should be a way to configure the thing, or a way to automate this.
If every day I get a new set of indexes, because logstash inserts data for services into indexes that do change their index name, every day, since it appends date at the end, there should be a way to control number of shards and replicas, otherwise, soon or later, indexes would start having default 1 replica unless we're manually intervening all the time... I'm sure I'm missing completelly the point, could you help me.

Thank you in advance.
Best regards!

[Christian_Dahlqvist]

This is typically managed through index templates, so I would recommend you have a look at that.

[alexolivan (Alejandro Olivan)]

Hi! thank you for your help...

Ok... readed that... a little bit overwheelming, when considering how simple is what it was, but here we go:

Provide that I have a bunch of logstash-servicefoo-YYYY.MM.DD, logstash-servicebar-YYYY.MM.DD, as were the result of folowing the classic ELK howtos of my era (1.x).
Provide that I don't care about aliasing, iteming or remaping and all that, but just want to have 2 replicas and lets say, 6 shards, for my minimal mockup cluster

Then I would need omething like that.... right?

curl -XPUT 'localhost:9200/_template/logstash?pretty' -H 'Content-Type: application/json' -d'
{
"template" : "logstash-*",
"settings" : {
"number_of_replicas" : 2,
"number_of_shards" : 6
}
}
'

I'm tryng to try it, but, pasting this into the console seems to not like the server probably there's a typo...

EDIT: worked as a one-liner...
curl -XPUT '192.168.100.4:9200/_template/logstash?pretty' -H 'Content-Type: application/json' -d'{ "template" : "logstash-*", "settings" : { "number_of_replicas" : 2, "number_of_shards" : 6 } }'

which caused:
{
"acknowledged" : true
}

Let's see what happens next!

EDIT: ouch! that screwed the default logstash template... I think I'm starting to get the point behind templates, and how it adds a whole new layer of possible configuration on incoming data... but I miss a litle bit of more classic control through config files, at least for defaults