Typical latency of bulk insert


#1

Hi,

This question may not have sufficient data to provide an accurate answer, but I'll settle for an approximate one. I'm planning to perform a lot of bulk inserts with node js elasticsearch, in chunks of 5MB. The ES is installed on a quad core, 8GB memory machine, and the script that runs the bulk inserts is located on another machine, but in the same VPN. I'm wondering what is the latency that I'm supposed to expect from each insert (50ms, 500ms, 50s?), since this value is important to ensure the proper flow of my script.

Thanks.


(Ashley Mannix) #2

It is one of those how long is a piece of string questions. The best way to answer your question would probably be to measure it in your environment and see. Are you talking about latency between submitting the bulk request and receiving confirmation, or submitting the bulk request and the results being observable to queries?


#3

I'm talking about latency between submitting the bulk request and receiving confirmation


(Christian Dahlqvist) #4

If you benchmark with different bulk sizes and varying number of concurrent writers, you will be able to determine the throughput your cluster can handle. Depending on how much data you send to Elasticsearch in parallel, the latency of each bulk request will vary as the amount it takes to process this volume depends on the maximum throughput the cluster can sustain. Smaller bulk requests and fewer writers will most likely result in lower latencies, but you may see the same total throughput with larger requests and longer latencies.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.