# Pipeline queue is getting filled and never able to control, using below drop filter plugin **URL:** \ **Category:** Logstash\ **Created:** [December 1, 2022, 7:18pm UTC](https://discuss.elastic.co/t/pipeline-queue-is-getting-filled-and-never-able-to-control-using-below-drop-filter-plugin/320287 "2022-12-01T19:18:28Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![rravitech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rravitech/32/108994_2.png) [@rravitech](https://discuss.elastic.co/u/rravitech)\ **Post date:** [December 1, 2022, 7:18pm UTC](https://discuss.elastic.co/t/pipeline-queue-is-getting-filled-and-never-able-to-control-using-below-drop-filter-plugin/320287/1 "2022-12-01T19:18:28Z")
I am trying to implement 2 pipelines, when one is about to get blocked, i am trying to drop events form that pipeline. So that the other one, which is half filled or healthy will keep receiving the events. - 1. Here in the below configuration i have es-host1 which is having 500MB queue size, and es-host2 which will eventually getting filled soon, since i kept the size with 50MB. - 1. So now when i flush logs of 2000/sec, i see that es-host2 is filling to its fullest first. - 1. When es-host1 is still not fully filled the output of es-host1 [eric-data-search-engine:9200] which also not receiving logs. - 1. I know that this is expected, according to design of logstash if any one of the queues is full, the others will be blocked. But in my case, i am trying to flush the log event in pipeline queue which is about to fill in my below config ```auto if ["queueSize" > 30000000] { // the size declared is 50MB but when it is reaching 30MB i am tryign to clear the log event by droping it drop { } } ``` ```auto apiVersion: v1 kind: ConfigMap metadata: name: logstash-configmap namespace: udp-test-poc data: logstash.yml: | http.host: "0.0.0.0" http.port: 9600 pipelines.yml: | - pipeline.id: intake queue.type: persisted queue.max_bytes: 500mb path.config: "/usr/share/logstash/config/logstash.conf" - pipeline.id: es-host1 queue.type: persisted queue.max_bytes: 500mb path.config: "/usr/share/logstash/config/es1.conf" - pipeline.id: es-host2 queue.type: persisted queue.max_bytes: 50mb path.config: "/usr/share/logstash/config/es2.conf" logstash.conf: | input { beats { port => 5044 } } output { pipeline { send_to => ["es-host1" , "es-host2"] } } es1.conf: | input { pipeline { address => "es-host1" } } filter { http { url => "http://localhost:9600/_node/stats/pipelines/es-host1" target_body => "[@metadata][stats]" target_headers => "[@metadata][statsHeaders]" add_field => {"queueSize" => "%{[@metadata][stats][pipelines][es-host1][queue][queue_size_in_bytes]}" } } mutate { convert => { "queueSize" => "integer" } } } output { elasticsearch { ilm_enabled => false hosts => ["eric-data-search-engine:9200"] user => 'logstash' password => '${LOGSTASH_PW}' index => "logstash-beta-%{+YYYY.MM.dd}" } } es2.conf: | input { pipeline { address => "es-host2" address => "es-host2" } } filter { http { url => "http://localhost:9600/_node/stats/pipelines/es-host2" target_body => "[@metadata][stats]" target_headers => "[@metadata][statsHeaders]" add_field => {"queueSize" => "%{[@metadata][stats][pipelines][es-host2][queue][queue_size_in_bytes]}" } } mutate { convert => { "queueSize" => "integer" } } if ["queueSize" > 30000000] { drop { } } } output { elasticsearch { ilm_enabled => false hosts => ["ext-se:9200"] user => 'logstash' password => '${LOGSTASH_PW}' index => "logstash-beta-%{+YYYY.MM.dd}" } } ``` When i query logstash:9600, i get this below data ```auto ------------------------------------------------- es2 "queue_size_in_bytes": 52431853, // which grows more than 30000000 "max_queue_size_in_bytes": 52428800 // 50MB of size es1 "queue_size_in_bytes": 52196678, // Still it has more size "max_queue_size_in_bytes": 524288000 //500MB of size ------------------------------------------------- ``` Here in my case, if I drop the events using the drop filter plugin, can i control the pipeline getting full. Or is my implementation on the filter plugin is wrong? please suggest. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 1, 2022, 9:43pm UTC](https://discuss.elastic.co/t/pipeline-queue-is-getting-filled-and-never-able-to-control-using-below-drop-filter-plugin/320287/2 "2022-12-01T21:43:58Z")
That's an interesting approach. I cannot explain why it does not work. I just wanted to mention another approach that drops data described [here](https://discuss.elastic.co/t/logstash-multiple-pipelines-failure-with-persistent-queues/296520/2). ---
**Author:** ![rravitech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rravitech/32/108994_2.png) [@rravitech](https://discuss.elastic.co/u/rravitech)\ **Post date:** [December 2, 2022, 5:01am UTC](https://discuss.elastic.co/t/pipeline-queue-is-getting-filled-and-never-able-to-control-using-below-drop-filter-plugin/320287/3 "2022-12-02T05:01:19Z")
@Badger I could see there is some limitations in UDP system, I was initially trying out this UDP system has limitation, where the log events with larger size are not able to get delivered. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 30, 2022, 5:02am UTC](https://discuss.elastic.co/t/pipeline-queue-is-getting-filled-and-never-able-to-control-using-below-drop-filter-plugin/320287/4 "2022-12-30T05:02:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.