We have a use case where we are inserting data in Elastic search cluster at 19-20K QPS. We are not getting any errors while inserting data using bulk processor.
However there are times where we observe read latency of about 5-10mins in Kibana(installed on a separate single client node). I think the cluster is properly scaled since writes are not giving any issues. Another behavior observed is that if the inserting application is started Kibana shows the latest data and the older data gets filled after some time. This essentially proves that data is not available for search.
We are using Elasticsearch version 2.4 and Kibana Version 4.6.2.
Any suggestions for the same. Thanks in advance.
The fact that the cluster is able to handle the write load does not necessarily mean that it is sized correctly. Searching and indexing use the same resources in the cluster in terms of CPU and disk I/O, so it could very well be that you do not currently have enough headroom available to be able to serve searches at an acceptable latency.
What does CPU and disk I/O look like during indexing? How does this change when you process a query through Kibana that is slow? What does the statistics for different visualisations in the dashboard show?
This probably points out that I/O is the bottleneck even when we are using AWS GP2 SSD EBS volumes. Are there any parameters which we can tweak to optimize the same?
What does CPU and disk I/O look like during indexing? You can get this e.g. via top and iostat. How does this change when you process a query through Kibana that is slow?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
