# Logstash received data but dont push forward to next hoop, data come into Logstash queue

**URL:** <https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056>\
**Category:** Logstash\
**Created:** [March 29, 2018, 8:53am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056 "2018-03-29T08:53:52Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![tatdat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tatdat/32/113160_2.png) [@tatdat](https://discuss.elastic.co/u/tatdat)\
**Post date:** [March 29, 2018, 8:53am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056/1 "2018-03-29T08:53:52Z")

</div>

Im using Elastic stack 6.2.2, Centos 7.  
LS-Collector have 8GB RAM, 8vPCU.  
Number event proccess : 4k e/s

My data flow :

Filebeat -\> LS-Collector -\> RabbitMQ -\> LS-Proccessor -\> ES

In LS-Collector, i use file output filter to debug. Enabled filebeat debug

In client, i try to make some log example, all log pushed to LS-Collector via filebeat (1)  
In LS-Collector, i check file output and i saw: sometime log was double, and logstash queue increase ..  
In RabbitMQ, i checked in RabbitMQ mamangement, no data in queue. Checked on Pipeline monitor (in Kiabana), number event proccessed \< all event sent by filebeat . Have same result when i check log in kibana. Not enought event/document from client (sent by filebeat) ...

Why did queue in LS-Collector increase? Why are some documents not pushed forward to RabbitMQ and LS-Processor from LS-Forwarder.

P/s: I'm using multiple pipeline in LS and make sure all event will be proccess by Logtash.

Thanks

---

<div class="post-metadata">

**Author:** ![tatdat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tatdat/32/113160_2.png) [@tatdat](https://discuss.elastic.co/u/tatdat)\
**Post date:** [March 30, 2018, 1:15am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056/2 "2018-03-30T01:15:37Z")

</div>

Someone help me ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 30, 2018, 6:19am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056/3 "2018-03-30T06:19:25Z")

</div>

It would be easier to help if you shared your Logstash config.

---

<div class="post-metadata">

**Author:** ![tatdat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tatdat/32/113160_2.png) [@tatdat](https://discuss.elastic.co/u/tatdat)\
**Post date:** [March 30, 2018, 8:26am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056/4 "2018-03-30T08:26:33Z")

</div>

Hi @Christian_Dahlqvist, all config or what? , in my LS-collector or LS-Proccessor. Because i have many config in logstash....

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 30, 2018, 8:59am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056/5 "2018-03-30T08:59:42Z")

</div>

If you do not get any data into RabbitMQ, I guess the issue might be with the LS-collector config(s), so that would probably be a good place to start.

---

<div class="post-metadata">

**Author:** ![tatdat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tatdat/32/113160_2.png) [@tatdat](https://discuss.elastic.co/u/tatdat)\
**Post date:** [March 30, 2018, 9:11am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056/6 "2018-03-30T09:11:43Z")

</div>

this is my config in LS-collector

```
input {
  beats {
    port => 5044
  }
}
filter {
  if [type] == "wineventlog" and [event_id] == 5156 {
    drop { }
  }
  mutate {
   add_field => {"beatname" => "%{[@metadata][beat]}"}
   add_field => {"beattype" => "%{[@metadata][type]}"}
   #add_field => {"beatversion" => "%{[@metadata][version]}"}
   #add_field => {"beatversion" => "%{[beat][version]}"}
  }
}

output {
    rabbitmq {
        exchange => "logstash"
        exchange_type => "direct"
        key => " **** -key"
        host => " ****"
        vhost => "elastic-stack"
        durable => true
        persistent => true
        port => *****
        user => " ****"
        password => " ****"

    }

    if [type] == "wineventlog" {
        if [event_id] == 4776 or [event_id] == 4732 or [event_id] == 4733 or [event_id] == 4728 or [event_id] == 4729 or [event_id] == 4756 or [event_id] == 4757 or [event_id] == 4624 or [event_id] == 4634 {
           udp {
              host => " ****"
              port => "514"
           }
        }

    }

  if [type] == "wifishop" {
    rabbitmq {
        exchange => "frt"
        exchange_type => "direct"
        key => "frt-key"
        host => " ****"
        vhost => " ****"
        durable => true
        persistent => true
        port => *****
        user => "***"
        password => " *****"
    }
    file {
        path => "/tmp/frt.log"
        codec => line
    }
  }
}

```

I dont get log (in rabbitmq queue) from this config but i saw log in file " **/tmp/frt.log**"

```
if [type] == "wifishop" {
        rabbitmq {
            exchange => "frt"
            exchange_type => "direct"
            key => "frt-key"
            host => " ****"
            vhost => " ****"
            durable => true
            persistent => true
            port => *****
            user => "***"
            password => " *****"
        }
        file {
            path => "/tmp/frt.log"
            codec => line
        }
      }

```

Check on Rabbitmq management

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/5/1582bcdbf733573ac33cd53b32a3b40385f9f3a8.jpg)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 30, 2018, 9:16am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056/7 "2018-03-30T09:16:47Z")

</div>

Does any input from Beats have type set to `wifishop`? It seems like RabbitMQ is receiving the `logstash` data.

---

<div class="post-metadata">

**Author:** ![tatdat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tatdat/32/113160_2.png) [@tatdat](https://discuss.elastic.co/u/tatdat)\
**Post date:** [March 30, 2018, 9:32am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056/8 "2018-03-30T09:32:14Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> Does any input from Beats have type set to wifishop?

Sure, because i checked in kibana, sometime, have some log with this type. but it not all log with ` wifishop` type.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2018, 9:32am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056/9 "2018-04-27T09:32:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.