Logstash - Rabbitmq - same message repeatedly being loaded

Elastic Stack Logstash
1

Hi There,

I have configured rabbitmq as a broker for reliability, below is my configuration:

input {
udp {
port => 5114
type => "syslog.imq"
}
}

output {

    if [type] == "syslog.imq" {

            rabbitmq {
                    key => "mykey"
                    exchange => "myex"
                    exchange_type => "direct"
                    user => "username"
                    password => "password"
                    host => "127.0.0.1"
                    port => 5672
                    durable => true
                    persistent => true
            }

    }

}

input {
rabbitmq {
host => "127.0.0.1"
queue => "myq"
durable => true
key => "mykey"
exchange => "myex"
threads => 1
prefetch_count => 50
port => 5672
user => "user"
password => "password"
type => "syslog.raw"
tags => "syslog.input"
tags => "ti.pending"
}
}

filter {

    if "syslog.input" in [tags] {

 # formatting commands

                          mutate {

                                    remove_tag => "syslog.input"
                                    add_tag => "syslog.output"
                            }


}

}

output {

    if "syslog.output" in [tags] {

                    elasticsearch{

                            hosts => "localhost:9200"
                            action => "index"
                            index => "myfile-%{+YYYY.MM.dd}"
                            template => "/etc/logstash/file1.json"
                            template_name => "myfile-*"
                            manage_template => "true"
                            template_overwrite => "true"
                    }
            }
    }

Problem: One message is being repeatedly sent to Elasticsearch.

There are no errors in logstash and elasticsearch log files.

Please help me.

Regards,

Jay

2

Is all of this in the same Logstash instance? Are you sure type => "syslog.raw" for the rabbitmq input overwrites the existing value?

3

Yes all of these are in same instance, I am really not sure how to ensure that Rabbitmq overwrites type = SYSLOG.IMQ. I would appreciate if you could give some hints.

I am building this to capture firewall SYSLOG events, using below workflow:

Firewall (TCP SYSLOG) -> NXLOG (TCP SSL) -> LS -> RMQ -> LS -> ES

Not sure if I need two instances of LS (Logstash), I am trying to avoid multiple LS instances being Java base.

Alternatively I could use Fluentd instead of first instance of LS and then submit to RMQ -> LS.

Please advise if there is any other solution.

Regards,

Ajay

4

I am really not sure how to ensure that Rabbitmq overwrites type = SYSLOG.IMQ.

Look at the events in Elasticsearch. What's their type?

Not sure if I need two instances of LS (Logstash), I am trying to avoid multiple LS instances being Java base.

Well, you either pay with RAM and CPU or with configuration complexity. Your pick.

5

Thank you.
I have created two instances of Logstash on same server to avoid any looping.
This is working fine.