Logstash received data but dont push forward to next hoop, data come into Logstash queue

Im using Elastic stack 6.2.2, Centos 7.
LS-Collector have 8GB RAM, 8vPCU.
Number event proccess : 4k e/s

My data flow :

Filebeat -> LS-Collector -> RabbitMQ -> LS-Proccessor -> ES

In LS-Collector, i use file output filter to debug. Enabled filebeat debug

In client, i try to make some log example, all log pushed to LS-Collector via filebeat (1)
In LS-Collector, i check file output and i saw: sometime log was double, and logstash queue increase ..
In RabbitMQ, i checked in RabbitMQ mamangement, no data in queue. Checked on Pipeline monitor (in Kiabana), number event proccessed < all event sent by filebeat . Have same result when i check log in kibana. Not enought event/document from client (sent by filebeat) ...

Why did queue in LS-Collector increase? Why are some documents not pushed forward to RabbitMQ and LS-Processor from LS-Forwarder.

P/s: I'm using multiple pipeline in LS and make sure all event will be proccess by Logtash.

Thanks

Someone help me ?

It would be easier to help if you shared your Logstash config.

Hi @Christian_Dahlqvist, all config or what? , in my LS-collector or LS-Proccessor. Because i have many config in logstash....

If you do not get any data into RabbitMQ, I guess the issue might be with the LS-collector config(s), so that would probably be a good place to start.

this is my config in LS-collector

input {
  beats {
    port => 5044
  }
}
filter {
  if [type] == "wineventlog" and [event_id] == 5156 {
    drop { }
  }
  mutate {
   add_field => {"beatname" => "%{[@metadata][beat]}"}
   add_field => {"beattype" => "%{[@metadata][type]}"}
   #add_field => {"beatversion" => "%{[@metadata][version]}"}
   #add_field => {"beatversion" => "%{[beat][version]}"}
  }
}

output {
    rabbitmq {
        exchange => "logstash"
        exchange_type => "direct"
        key => "****-key"
        host => "****"
        vhost => "elastic-stack"
        durable => true
        persistent => true
        port => *****
        user => "****"
        password => "****"

    }

    if [type] == "wineventlog" {
        if [event_id] == 4776 or [event_id] == 4732 or [event_id] == 4733 or [event_id] == 4728 or [event_id] == 4729 or [event_id] == 4756 or [event_id] == 4757 or [event_id] == 4624 or [event_id] == 4634 {
           udp {
              host => "****"
              port => "514"
           }
        }

    }

  if [type] == "wifishop" {
    rabbitmq {
        exchange => "frt"
        exchange_type => "direct"
        key => "frt-key"
        host => "****"
        vhost => "****"
        durable => true
        persistent => true
        port => *****
        user => "***"
        password => "*****"
    }
    file {
        path => "/tmp/frt.log"
        codec => line
    }
  }
}

I dont get log (in rabbitmq queue) from this config but i saw log in file "/tmp/frt.log"

if [type] == "wifishop" {
        rabbitmq {
            exchange => "frt"
            exchange_type => "direct"
            key => "frt-key"
            host => "****"
            vhost => "****"
            durable => true
            persistent => true
            port => *****
            user => "***"
            password => "*****"
        }
        file {
            path => "/tmp/frt.log"
            codec => line
        }
      }

Check on Rabbitmq management

Does any input from Beats have type set to wifishop? It seems like RabbitMQ is receiving the logstash data.

Sure, because i checked in kibana, sometime, have some log with this type. but it not all log with wifishop type.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.