I currently ingest data into my Elasticsearch cluster from four sources in the first 15 minutes of each hour. During these 15 minutes, queries are slower and performance is impacted.
Are there any alternatives that could mitigate this issue? Could the issue be that I am trying to ingest too much data at once?
Thank you for the help!
Thanks so much for reaching out, @echan23. I have a few follow up questions here:
What type of data are you ingesting? Why are you not ingesting continously?
Have yopu looked at the guide for optimizing indexing speed?
If indexing is affecting querying latencies if often means there is a resource constraint within the cluster. As indexing is I/O intensive it may be that the storage is not fast enough for the combined load you are putting the cluster under. It could also be some other resource that is the limiting factor. It would probably help if you provided some details around the size, hardware and configuration of your cluster and the load it is under when you see issues.
It is also always a good idea to specify which version of Elasticsearch you are using.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.