Hi,
In our logging clusters we're indexing at around 40k msg/sec and 500gb per index per day. The storage backend is ceph and sometimes it struggles as load increases over time etc.
Consequently, I'm toying wondering what I might do to optimize indexing in a way that accounts for sub-optimal storage. One of those things I'm considering is to double the size of index.translog.flush_threshold_size
. Is this considered a good move when looking to maximise throughput? We're already have index.refresh_interval
set to 30s so I figured making this change could align reasonably well with that...
Regards,
D