We have a large-ish cluster used for log search, document size usually a few hundred bytes max.
We have had to re-spin nodes, causing recoveries and rebalances.
During rebalance, and sometimes at points in the recovery phase, indexing slows to the point where log lines get dropped (since the senders are configured to drop rather than queue).
Is this indexing slowdown expected? Is there any remedy for this other than set up Kafka (or Logstash 2.0)?
We are using AWS, 20 i2.8xlarge instances split across two availability zones. We have rebalancing throttled to 2 streams and recovery to 8. Would appreciate any insight into what's going on here.