Logstash received data but dont push forward to next hoop, data come into Logstash queue

Elastic Stack Logstash
1

Im using Elastic stack 6.2.2, Centos 7.
LS-Collector have 8GB RAM, 8vPCU.
Number event proccess : 4k e/s

My data flow :

Filebeat -> LS-Collector -> RabbitMQ -> LS-Proccessor -> ES

In LS-Collector, i use file output filter to debug. Enabled filebeat debug

In client, i try to make some log example, all log pushed to LS-Collector via filebeat (1)
In LS-Collector, i check file output and i saw: sometime log was double, and logstash queue increase ..
In RabbitMQ, i checked in RabbitMQ mamangement, no data in queue. Checked on Pipeline monitor (in Kiabana), number event proccessed < all event sent by filebeat . Have same result when i check log in kibana. Not enought event/document from client (sent by filebeat) ...

Why did queue in LS-Collector increase? Why are some documents not pushed forward to RabbitMQ and LS-Processor from LS-Forwarder.

P/s: I'm using multiple pipeline in LS and make sure all event will be proccess by Logtash.

Thanks

2

Someone help me ?

3

It would be easier to help if you shared your Logstash config.

4

Hi @Christian_Dahlqvist, all config or what? , in my LS-collector or LS-Proccessor. Because i have many config in logstash....

5

If you do not get any data into RabbitMQ, I guess the issue might be with the LS-collector config(s), so that would probably be a good place to start.

6

this is my config in LS-collector

input {
  beats {
    port => 5044
  }
}
filter {
  if [type] == "wineventlog" and [event_id] == 5156 {
    drop { }
  }
  mutate {
   add_field => {"beatname" => "%{[@metadata][beat]}"}
   add_field => {"beattype" => "%{[@metadata][type]}"}
   #add_field => {"beatversion" => "%{[@metadata][version]}"}
   #add_field => {"beatversion" => "%{[beat][version]}"}
  }
}

output {
    rabbitmq {
        exchange => "logstash"
        exchange_type => "direct"
        key => "****-key"
        host => "****"
        vhost => "elastic-stack"
        durable => true
        persistent => true
        port => *****
        user => "****"
        password => "****"

    }

    if [type] == "wineventlog" {
        if [event_id] == 4776 or [event_id] == 4732 or [event_id] == 4733 or [event_id] == 4728 or [event_id] == 4729 or [event_id] == 4756 or [event_id] == 4757 or [event_id] == 4624 or [event_id] == 4634 {
           udp {
              host => "****"
              port => "514"
           }
        }

    }

  if [type] == "wifishop" {
    rabbitmq {
        exchange => "frt"
        exchange_type => "direct"
        key => "frt-key"
        host => "****"
        vhost => "****"
        durable => true
        persistent => true
        port => *****
        user => "***"
        password => "*****"
    }
    file {
        path => "/tmp/frt.log"
        codec => line
    }
  }
}

I dont get log (in rabbitmq queue) from this config but i saw log in file "/tmp/frt.log"

if [type] == "wifishop" {
        rabbitmq {
            exchange => "frt"
            exchange_type => "direct"
            key => "frt-key"
            host => "****"
            vhost => "****"
            durable => true
            persistent => true
            port => *****
            user => "***"
            password => "*****"
        }
        file {
            path => "/tmp/frt.log"
            codec => line
        }
      }

Check on Rabbitmq management

image
image.jpg921×503 77 KB

7

Does any input from Beats have type set to wifishop? It seems like RabbitMQ is receiving the logstash data.

8

Sure, because i checked in kibana, sometime, have some log with this type. but it not all log with wifishop type.