# Regarding Bulk Indexing Requests **URL:** \ **Category:** Elasticsearch\ **Created:** [April 9, 2024, 6:56am UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034 "2024-04-09T06:56:06Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 9, 2024, 6:56am UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/1 "2024-04-09T06:56:06Z")
Hi, Is there a way I can view the size (number of documents) of a bulk index request once it hits the elasticsearch queue? I want to estimate how efficiently logstash is batching requests that it pushes to elasticsearch. Thx D ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 9, 2024, 7:09am UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/2 "2024-04-09T07:09:38Z")
No, I do not think there is. Logstash works with a specific max [pipeline batch size](https://www.elastic.co/guide/en/logstash/8.13/tuning-logstash.html#tuning-logstash) (defaults to 125) and this is as far as I know the largest bulk size that will be sent. If you have multiple elasticsearch ouput plugins configured, controlled by conditionals, it is possible that bulk sizes by default will be much smaller which can indeed be inefficient. ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 9, 2024, 7:29am UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/3 "2024-04-09T07:29:59Z")
So how can we measure our pipeline efficiency? Sending salami sized bulks to elasticsearch can make it look like elasticsearch is the bottleneck when it may not be. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 9, 2024, 7:42am UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/4 "2024-04-09T07:42:20Z")
I would recommend the following: - Try to have reasonably focused pipelines with a single Elasticsearch output plugin each. Use [multiple defined pipelines](https://www.elastic.co/guide/en/logstash/8.13/multiple-pipelines.html) together with [pipeline to pipeline communication](https://www.elastic.co/guide/en/logstash/8.13/pipeline-to-pipeline.html) to control the flow. Avoid having large number of different Elasticsearch outputs controlled by conditionals. If you think you need the Elasticsearch output to send data to multiple indices, store the index name in the metadata and use this in a single Elasticsearch output. - Increase the batch size to something slightly larger, e.g. 1000 or 2000. If you have focused pipelines there is rarely a need to go much higher. - Use Logstash monitoring metrics to see how much data is processed by different plugins. - Have a look at the [Logstash performance tuning and monitoring guide](https://www.elastic.co/guide/en/logstash/8.13/performance-tuning.html) if you have not already. ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 10, 2024, 10:44am UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/5 "2024-04-10T10:44:16Z")
Am I correct in thinking that reading messages from message queues will saturate the logstash pipeline more effectively that having large numbers of beats, or any other agent types, fire their output directly at the input port? And, assuming a sane config, will therefore size bulk writes more efficiently? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 10, 2024, 10:50am UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/6 "2024-04-10T10:50:55Z")
Reading messages from a large message queue is likely to result in full batches of events being processed. I am not sure how the input plugins receiving data from Beats map this to batches, so am not sure whether it would have any impact. That said I do not think it would hust performance. ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 10, 2024, 10:54am UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/7 "2024-04-10T10:54:53Z")
I say this because I've seen much better scaling on the backend when reading from queues (pull) vs receiving directly from agents (push). But my thinking is being disputed internally because there aren't metrics which directly corroborate my stance. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 10, 2024, 11:56am UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/8 "2024-04-10T11:56:11Z")
I have not run any tests or benchmarks on it so do not know for sure. I do suspect using a message queue would help in a number of ways if you have very large number of Beats feeding data, e.g. more evently distributing load and added resiliency. If the pipelines that are receiving data are inefficient (as per initial notes), adding a message queue is unlikely to improve performance much by itself. ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 10, 2024, 12:53pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/9 "2024-04-10T12:53:38Z")
Our pipelines are set up in the manner you recommend. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 10, 2024, 12:56pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/10 "2024-04-10T12:56:20Z")
Have you overridden the default batch size? ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 10, 2024, 12:59pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/11 "2024-04-10T12:59:37Z")
Yes. We also run extra workers. Should we consider increasing the batch delay? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 10, 2024, 1:05pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/12 "2024-04-10T13:05:49Z")
> [@dawiro](#): > > Should we consider increasing the batch delay? Nor sure that will help much. Lets take a step back first. How have you determined that Elasticsearch is not the limiting factor? What indexing throughput are you seeing? How many indices and shards are you actively indexing into? What is the size and specification of your cluster? What kind of storage are you using? ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 10, 2024, 1:17pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/13 "2024-04-10T13:17:45Z")
Elasticsearch isn't the limiting factor because we've run tests where for a given fixed topology (elasticsearch/logstash) we see higher sustained throughput with no loss when reading from queues. We see loss when pushing and ingestion rate is more erratic. Logstash isn't seeing 429s from the backend. The shard layout is set at one primary and one replica per node. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 10, 2024, 1:23pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/14 "2024-04-10T13:23:49Z")
OK. Good to see that you have taken a systematic approach. Sounds like a reasonable conclusion. What does your pipelines look like? Have you looked at pipeline performance statistics to see if you have any slow/expensive filters configured? ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 10, 2024, 1:31pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/15 "2024-04-10T13:31:25Z")
That is a next step to take. ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 10, 2024, 1:33pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/16 "2024-04-10T13:33:46Z")
How can we determine, aside from cpu/heap utilisation, how/why logstash is choking on the frontend? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 10, 2024, 1:43pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/17 "2024-04-10T13:43:21Z")
> [@dawiro](#): > > The shard layout is set at one primary and one replica per node. How many indices are you actively indexing into? Is it just one? > [@Christian\_Dahlqvist](#): > > What is the size and specification of your cluster? What kind of storage are you using? What indexing throughput are you seeing? With respect to tuning Logstash I would look at performance metrics and try to identifying inefficient filters. I have on numerous occasions seen e.g. Inefficient grok filters slow down throughput a lot. There are also filetrs that rely on callouts that can severely limit throughput. ---
**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Post date:** [April 10, 2024, 1:55pm UTC](https://discuss.elastic.co/t/regarding-bulk-indexing-requests/357034/18 "2024-04-10T13:55:42Z")
Yes, just a single index. We don't actually do any heavy transforms on our data. We try to keep it relatively simple - just index the data as smoothly as possible and leave data formatting to the upstream.