Can someone point me to info on configure elasticsearch to roll over indice from daily to hourly?
Using elasticsearch 6.2.4. The current index is created from fluentd (logstash_format=true) with timestring '%Y.%m.%d'.
Thanks.
Do you have enough data volumes to warrant this?
If you don't then you're going to be wasting a fair bit of resources on shards is all.
Yes there is enough data (7-10 GB) to at the least investigate the give and take. I would like to configure the rollover to hourly and determine if hourly or daily.
That's not really worth it, you're going to be way over-sharded.
Hourly indices can work if you have very large data volumes (multiple TB/day) and relatively short retention periods. As described in this blog post having lots of small indices and shards can be very inefficient, but each separate index also carries additional overhead in terms of mappings, that need to be stored on the cluster state.
That said, if you are using index names with a date string in the name, this is generally controlled by the application writing to Elasticsearch, in this case FluentD, so that is where you would need to configure it. It would be different if you were using the rollover index API.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.