The amount of log data we have has recently jumped up to ~700000 rows per minute, and our elasticsearch cluster is no longer able to process the index requests quickly enough(takes about 10 minutes for 1 minute of data).
Our current setup had 5 shards on 5 nodes with 8 cores and 56GB of RAM each. I've tried increasing the shard size to 10 and adding a couple more nodes with no noticeable improvement.
Here are the settings I've tried tweaking so far:
http.max_content_length: 1000mb
indices.memory.index_buffer_size: 30%
indices.memory.min_shard_index_buffer_size: 12mb
indices.memory.min_index_buffer_size: 96mb
index.refresh_interval: 15s
Any important settings I'm missing? Do we need more nodes/shards?
Thanks!
Evan