Multiple index parameters per index type

Elastic Stack Elasticsearch
1

I have multiple type of index files generated by different data sources. They all reside on the same cluster.

index-typeA
index-typeB
index-typeC

each one has different shard count, replica count requirements.

is there a way in my elasticsearch.yaml to specify

index.number_of_shards
index.number_of_replicas

parameters for each specific index? Or this pertains to the cluster itself?

I understand that you can do this by API for each index, but wont' this be considered as transient setting and will be reset in case full cluster restart?

Thanks in advance.

2

Anything in elasticsearch.yml applies to the cluster. If you want to apply different ones to different indices then you should use a template :slight_smile:

Only for replicas can you do this dynamically. You can specify number of shards only when you create the index.

3

Ok so at the moment I have my index mappings generated dynamically. If I use JSON over http, something like:

curl -XPUT 'http://localhost:9200/lwes-mm-dd-yyyy/' -d '
index :
number_of_shards : 5
number_of_replicas : 1
'

is there a way to generate the date dynamically ?
Also do I need to tie to something like a crontab to execute in daily basis?

4

Look into templates instead, they're much better.

5

I want to, but I guess I am not understanding how the template is being executed by the elasticsearch. In my case when I have rolling log, how do I tell elastic search to run a specific template everyday.

6

When you create the template you define a pattern that it should match, so when an index is created it looks for templates that match and then applies them.

7

Ok I think I got the handle of it

curl -XGET "http://localhost:9200/_template/template_lwes?pretty"
{
"template_lwes" : {
"order" : 0,
"template" : "lwes-*",
"settings" : {
"index" : {
"number_of_shards" : "5",
"number_of_replicas" : "1"
}
},
"mappings" : { },
"aliases" : { }
}
}

I was able to apply the template. So if I didn't assign any mappings, I am assuming that dynamic ones will be applied, correct?

8

That is correct.

9

Thanks for the help