Rotate index every 6 hours

Elastic Stack Logstash
#1

Hi

i have 2 daily indices that are about 30-40Gb every day and even spread by 6 shards are big. When i have to restart one node, they take a long time to recover.

So i want to rotate then not by day, but by every 6 hours ... how can i do that? %{+YYYY.MM.dd} is daily, %{+YYYY.MM.dd-HH} is hour, how can i do a %{YYYY.MM.dd-H%6} ?

maybe i could use elasticsearch _rollover, but how to use it? i found no info about adding this to the templates

thanks in advance
daniel

#2

I think you are oversharding with this strategy.

That's only ~6GB, it's not large.

#3

Sorry, actually is 3 shards and 1 replica.
the problem i'm trying to solve is the cluster recovery time, with 2000 indices and around 13k shards takes between 1h to 3 hours to recover... and bigger indices are the ones that take longer.

by breaking those big indexes in 4 smaller shards, i hope that ES will recover then faster, as at least 3 of then didn't change and full recovering the last one would be faster as it is much smaller

#4

Having a very large number of shards to move around can also have a negative impact on recovery. 6GB is generally not very large, so I would probably not go smaller than that. Recovery might actually be helped by increasing the size rather than decreasing it. The only way to find out for sure is probably to test it though.

#5

i have tested! :smiley:

one 10GB index that is being updated takes about 15min
4*3GB indexes that only the last one is being updated take about 5min

the 3 indexes that are not being updated sync very fast and only the one that is really being updated takes longer

I know that faster disks and networks would make this problem a minor problem, but until i have then, i'm trying to finetune the current setup

#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.