Increase shards and replicas in new Index

[mrashid (MBR)]

In my logstash configuration I am creating Index daily with date using the below configuration :

 elasticsearch {
hosts => ["elkhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}-failurelogs"
document_type => "%{[@metadata][type]}" 

Now I want to create the index with higher shards for better memory utilization.

How to configure index creation with higher shards and replicas on a daliy basis from the logstash configuration ?

[warkolm (Mark Walkom)]

@mrashid

Now I want to create the index with higher shards for better memory utilization.

What do you mean by that? What problem are you trying to solve?

[mrashid (MBR)]

My assumption was that if i create more shards and segregate the events into the shards , better memory utilization will be possible and in turn the serach results will be faster.

I came across a similar issue here : Elasticsearch querying is terribly slow

However , now I believe that because the amount of data in my case is 2 GB to max 12GB , I am better of maintaining 3 shards and increase the heap max /min in elastic search to 40 gb.

[Christian_Dahlqvist]

If you have 2GB of data per day and only keep a few days you do only need a single primary shard per daily index and likely no more than a 4GB heap.

[warkolm (Mark Walkom)]

@mrashid

My assumption was that if i create more shards and segregate the events into the shards , better memory utilization will be possible and in turn the serach results will be faster.

So your searches are slow?