# Controlling elasticsearch \_bulk queue size **URL:** \ **Category:** Elasticsearch\ **Created:** [September 5, 2017, 2:48pm UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455 "2017-09-05T14:48:03Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Quintus](https://avatars.discourse-cdn.com/v4/letter/q/49beb7/32.png) [@Quintus](https://discuss.elastic.co/u/Quintus)\ **Post date:** [September 5, 2017, 2:48pm UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/1 "2017-09-05T14:48:03Z")
I've setup elasricsearch on a machine with the following specs: Storage: 500 GB HDD Ram: 8 GB CPU : Intel i7 Quad-Core On the same machine I have an application that listens to netflow data and exports it to elasticsearch. I've noticed , however , that the other application complains about the data being dropped since the bull queue is too long. I've tried increasing the bull queue size to something like 10000 bit it doesn't seem to help. New indices are build daily. Is there anyway to troubleshoot this ? How can I find out if elasricsearch is actually dropping the data ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 5, 2017, 2:50pm UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/2 "2017-09-05T14:50:28Z")
How many indices/shards are you actively indexing into? How many indexing threads do you have? ---
**Author:** ![Quintus](https://avatars.discourse-cdn.com/v4/letter/q/49beb7/32.png) [@Quintus](https://discuss.elastic.co/u/Quintus)\ **Post date:** [September 9, 2017, 8:25am UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/3 "2017-09-09T08:25:36Z")
Hi Christian, Sorry I thought I'd get e-mails for topics I'm watching. I haven't changed any of the default settings. I have one index created per day "ntopng-%y-%m-%d-\*" and I guess I have the default settings in place (so 5 shards)? 1. Is there a way I can find the right numbers? 2. what options do I have to resolve it? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 11, 2017, 5:34am UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/4 "2017-09-11T05:34:57Z")
Do you have monitoring installed? How many client connections are indexing in parallel? ---
**Author:** ![Quintus](https://avatars.discourse-cdn.com/v4/letter/q/49beb7/32.png) [@Quintus](https://discuss.elastic.co/u/Quintus)\ **Post date:** [September 11, 2017, 2:12pm UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/5 "2017-09-11T14:12:22Z")
I have one application currently communication with elasticsearch and it's currently on the same host. I just did a netatat and I huge number of connection to localhost:9200 .. could this be the reason ? I do not have any sort of monitoring installed ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 11, 2017, 2:45pm UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/6 "2017-09-11T14:45:13Z")
If you are indexing over too many connections in parallel, you will fill up the queues. In order to find the ideal number of indexing connections, start with a few connections and then gradually increase until the indexing throughput stops increasing or you start seeing bulk rejections. ---
**Author:** ![Quintus](https://avatars.discourse-cdn.com/v4/letter/q/49beb7/32.png) [@Quintus](https://discuss.elastic.co/u/Quintus)\ **Post date:** [September 11, 2017, 3:01pm UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/7 "2017-09-11T15:01:07Z")
Thanks .. I'll check this right away. How can I control the connections over elasticsearch? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 11, 2017, 3:02pm UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/8 "2017-09-11T15:02:07Z")
That depends on the application that is writing data to Elasticsearch. ---
**Author:** ![Quintus](https://avatars.discourse-cdn.com/v4/letter/q/49beb7/32.png) [@Quintus](https://discuss.elastic.co/u/Quintus)\ **Post date:** [September 12, 2017, 7:39pm UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/9 "2017-09-12T19:39:26Z")
one more question. Say if I can't control the application, what do I need to do with elasticsearch to handle this? Do I need something like SSD instead of HDD? or do i need to setup more indices? I'm just trying to figure out how elasticsearch works in this case Here's a reference to what I'm talking about: [https://github.com/ntop/ntopng/issues/645](https://github.com/ntop/ntopng/issues/645) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 10, 2017, 7:39pm UTC](https://discuss.elastic.co/t/controlling-elasticsearch--bulk-queue-size/99455/10 "2017-10-10T19:39:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.