In a spinning drive set up, we've seen the performance go down 8X after replication is enabled. This is too much a hit for us. I've read a published performance tip that recommends turning off replication then then it back on to avoid duplication indexing. But we would really love to be able to continuously replicate our data.
Our set up is two instances on spinning drives. The IO wait is maxed and appears to be the bottleneck. With duplicate indexing we'd expect to close to 2X penalty but any explaination why it's 8X?
To measure the system throughput, we are driving a known set of events through filebeat collectors, and then observing the elasticsearch performance. When the system is "healthy" the elasticsearch will be able to index the event traffic as it comes in. When the performance limit is reached, the indexing falls behind the event traffic, and events back up in the input queue.
With replication disabled (replication=0), we are ale to drive 8x the number of events vs when we have replication set to 1.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.