Writing to 2 different locations in S3

[mateo41 (Matt Rodriguez)]

Hi folks,

I'm using logstash to read from 2 rabbitMQ queues and write to two different locations in S3. Data from one queue will go to one location in S3, and data from the other queue will go to a different location in S3. I'm having difficulty configuring this. I realize that I need to use conditionals to set the appropriate output plugin, but the conditional is not correct and all of the data is written to one location in S3.

Here is the structure of my configuration file. I've omitted details about the queues and S3 locations for clarity. It seems like the conditional is always evaluating to false, because I keep writing to the output location specified in the else clause.

Any insight on how to fix this, or any techniques for testing the configuration, would be greatly appreciated.

Thanks,
Matt

Here's my abbreviated configuration
input {
rabbitmq {
id => "rabbitmq_logstash_ingest"
queue => "test-logstash"
}
rabbitmq {
id => "rabbitmq_logstash_ingest"
queue => "test-logstash2"
}
}
filter {
if [rabbitmq][queue] == "test-logstash" {
metrics {
meter => "events"
add_tag => "metric"
}
}
else {
metrics {
meter => "events"
add_tag => "metric2"
}
}
}
output {
if [rabbitmq][queue] == "test-logstash" {
s3 {
bucket => "data-test-bucket"
id => "rabbitmq_logstash_s3_output"
prefix => "logstash-throughput/%{+YYYYMMdd_HH}"
}
} else {
s3 {
bucket => "samba-eurozone-data-test"
id => "rabbitmq_logstash_s3_output2"
prefix => "logstash-throughput2/%{+YYYYMMdd_HH}"
}
}
}

[warkolm (Mark Walkom)]

Add a stdout so you can see the structure of the events.

But at a quick glance, I don't think you need the [rabbitmq] part as the field should just be called queue.

[mateo41 (Matt Rodriguez)]

Thank you.

I was able to solve this. Let explain what I did in case somebody else runs into a similar problem.

1. I enabled metadata in the rabbitmq input plugin like this:

metadata_enabled => true

2. I used a conditional in the filter plugin to add a tag like this

if [@metadata][rabbitmq_properties][routing-key] == "queue_name" {

    mutate {
        add_tag => "queue_name"
    }

3. I checked the tags in a conditional in the output plugins like this:

if "queue_name" in [tags] {
-- s3 output definition --
}

While I was figuring this out I added stdout to the output section, which helped me debug the issue.

Thanks,
Matt