The amount of log data we have has recently jumped up to ~700000 rows per minute, and our elasticsearch cluster is no longer able to process the index requests quickly enough(takes about 10 minutes for 1 minute of data).
Our current setup had 5 shards on 5 nodes with 8 cores and 56GB of RAM each. I've tried increasing the shard size to 10 and adding a couple more nodes with no noticeable improvement.
Here are the settings I've tried tweaking so far:
Any important settings I'm missing? Do we need more nodes/shards?