I am having Elasticsearch version 5.6.3 running on RHEL 7.4. This Cluster is of 10 nodes.
Hardware : 8 CPU and 32 GB RAM.
Hard Disk : HDD { CEPH Storage }
Elastic Data Paths : Multiple { 7 Different mount points }.
We are logging logs data into this ES cluster.
When we are performing IO Test using "fio" tool , we are getting throughput of sequential write to multiple mount points as below WRITE: bw=20.3MiB/s (21.3MB/s), 20.3MiB/s-20.3MiB/s (21.3MB/s-21.3MB/s), io=24.0GiB (25.8GB), run=1210725-1210725msec
When indexing is being performed , we are getting speed of 30-40 records are documented in indices per 10 seconds. Its very slow.
I would like to understand why is the difference so huge when its comes to realistic data pushing.
We don't support Elasticsearch with distributed filesystems, so it's unlikely you will find anything to fix your issue other than moving away from ceph sorry.
The issue is not multiple data paths, the issue is ceph is distributed.
You are then running a high performance distributed system on top of a distributed filesystem. It's not really designed for high performance.
Elasticsearch does generally not write very large segments sequentially, so your fio load is likely not very representative. I would recommend trying with a random read/write load instead as I would expect that to better match an Elasticsearch load pattern.
Could you please elaborate Large segments means ==> Single Document of huge mapping and complex structure ? OR in parallel pushing multiple documents to different indices ?
In our case , out mapping is not that huge and majorly consist of text. We do push multiple docs to multiple indices in parallel at a time from different applications of ours.
Also from Elasticsearch documentation , I learned that ES writes data in sequential manner. Hence I am doing sequential testing .. using MMAP as ioengine.
Also we have observed , CEPH storage having SSD are 50 times faster then CEPH HDD... both being on CEPH are distributed storages. Hence not able to reach conclusion by would HDD have such a slow IO when fio shows its IOPS = 3000-3500 and having bandwidth of 20Mibs ....
I find that dip benchmarks with large sequential reads and writes are not representative of normal Elasticsearch load. If you believe differently I would recommend setting up a cluster with fast local storage and run a test with representative load and profile io access patterns.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.