This question may not have sufficient data to provide an accurate answer, but I'll settle for an approximate one. I'm planning to perform a lot of bulk inserts with node js elasticsearch, in chunks of 5MB. The ES is installed on a quad core, 8GB memory machine, and the script that runs the bulk inserts is located on another machine, but in the same VPN. I'm wondering what is the latency that I'm supposed to expect from each insert (50ms, 500ms, 50s?), since this value is important to ensure the proper flow of my script.
It is one of those how long is a piece of string questions. The best way to answer your question would probably be to measure it in your environment and see. Are you talking about latency between submitting the bulk request and receiving confirmation, or submitting the bulk request and the results being observable to queries?
If you benchmark with different bulk sizes and varying number of concurrent writers, you will be able to determine the throughput your cluster can handle. Depending on how much data you send to Elasticsearch in parallel, the latency of each bulk request will vary as the amount it takes to process this volume depends on the maximum throughput the cluster can sustain. Smaller bulk requests and fewer writers will most likely result in lower latencies, but you may see the same total throughput with larger requests and longer latencies.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.