Daily index and monthly index query performance difference

Elastic Stack Elasticsearch
1

Hi,

I'm wondering the performance difference between daily monthly indices. Lets say i have an index named test and logstash output is configured like this:

output {
    elasticsearch {
        hosts => ["esnode1:9200"]
        index => "test-%{+YYYY.MM.dd}"
    }
}

it has 1 primary and 1 replica shard which are lets say 50mb each. so give or take 100mb for the daily indice. And I keep the logs for a month so 30 indices or 60 shards per month just for this index which at the end of the month adds up to around 3gb. What will happen differently if i change the logstash date to {YYYY.MM} and get 2 shards per month instead of 60? Will reading performance increase? Or because it's a significantly larger index than daily ones the reading performance decrease? Please assume these two questions being asked for a "last 30 days" search in Kibana.
I know it is always better to minimize the shard count in a cluster but do you sacrifice some things to achieve low shard count? Is there any drawback? I'd really like some clarification on this subject because I'm trying to figure out the best solution for my current environment before it bursts so any insight or tips is greatly appreciated.

Thanks in advance.

PS: I wasn't sure which segment i should ask this in, if elasticsearch is the wrong place to ask this please inform me on where to post it (Logstash, Kibana, etc)

2

3GB is still a rather small shard/index size so I would not expect any issues with query performance with monthly indices.

3

Hi @Christian_Dalhqvist, thanks for the quick reply.
Yes considering the recommended shard size is about 50gb, but what happens if i expand this 3gb scenario to 50 monthly indexes that are about the same size and around 10 indexes on daily config that are so large that you can't apply monthly or weekly configs?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.