LS Indexer not grabbing messages from Rabbitmq

Elastic Stack Logstash
1

My LS indexer won't get the messages in the rabbitmq queue. I can see that there are messages in the queue but they're not going anywhere.

My Logstash server is setup as ... LS shipper -> rabbitmq -> LS indexer

LS shipper conf:
output {
rabbitmq {
exchange => "logstash-rabbitmq"
exchange_type => "direct"
host => "127.0.0.1"
}
}

LS indexer conf:
input {
rabbitmq {
host => "127.0.0.1"
exchange => "logstash-rabbitmq"
}
}

Listing bindings ...
exchange amq.gen-CR4ZuvqzvxTAKk5kG4g4jw queue amq.gen-CR4ZuvqzvxTAKk5kG4g4jw []
exchange amq.gen-EHvne9rbXQviQTbni3jQQw queue amq.gen-EHvne9rbXQviQTbni3jQQw []
exchange amq.gen-ZSSjvqKT3C0g0HmZ1FSrfg queue amq.gen-ZSSjvqKT3C0g0HmZ1FSrfg []
exchange amq.gen-_FnraIfGrZBmM3A-I56bAg queue amq.gen-_FnraIfGrZBmM3A-I56bAg []
exchange amq.gen-cUA9eDp8wdDWQkjXGRkV4Q queue amq.gen-cUA9eDp8wdDWQkjXGRkV4Q []
exchange amq.gen-du9yHkP0NJLhBovlgIjRug queue amq.gen-du9yHkP0NJLhBovlgIjRug []
exchange amq.gen-uohpEtA8h0YeBtorHsCCYA queue amq.gen-uohpEtA8h0YeBtorHsCCYA []
exchange logstash-queue queue logstash-queue []
logstash-rabbitmq exchange amq.gen-CR4ZuvqzvxTAKk5kG4g4jw queue logstash []
logstash-rabbitmq exchange amq.gen-EHvne9rbXQviQTbni3jQQw queue logstash []
logstash-rabbitmq exchange amq.gen-ZSSjvqKT3C0g0HmZ1FSrfg queue logstash []
logstash-rabbitmq exchange amq.gen-cUA9eDp8wdDWQkjXGRkV4Q queue logstash []
logstash-rabbitmq exchange amq.gen-du9yHkP0NJLhBovlgIjRug queue logstash []
logstash-rabbitmq exchange amq.gen-uohpEtA8h0YeBtorHsCCYA queue logstash []
logstash-rabbitmq exchange logstash-queue queue logstash []

Listing queues ...
amq.gen-uohpEtA8h0YeBtorHsCCYA 0
amq.gen-EHvne9rbXQviQTbni3jQQw 0
amq.gen-_FnraIfGrZBmM3A-I56bAg 0
amq.gen-du9yHkP0NJLhBovlgIjRug 52
amq.gen-ZSSjvqKT3C0g0HmZ1FSrfg 0
amq.gen-CR4ZuvqzvxTAKk5kG4g4jw 3108
logstash-queue 0
amq.gen-cUA9eDp8wdDWQkjXGRkV4Q 0

I've restarted my LS indexer a few time, so there are some empty queues.

Any thoughts on what to check or what I'm missing?

Thanks

2

I believe a random queue is created if the queue isn't specified in the input. I might be wrong. But here's my working configuration with the same exact setup as yours. Hope it helps.

output {
rabbitmq {
exchange => "rabbitmq"
exchange_type => "direct"
host => "x.x.x.x"
durable => true
persistent => true
}
}

input {
rabbitmq {
queue => "rabbitmq-queue"
exchange => "rabbitmq"
host => "x.x.x.x"
durable => true
}
}

This is what it was before I changed to SSL.

3

Thanks for your help.

I'm not sure what the actual problem was, but it's working now. I think one issue was that rabbitmq thought that my output LS was rabbitmq producer and was expecting it to put messages in the exchange instead of taking them out of the queue. ... although at one time I was able to see that the output LS was a consumer, but bound to a new queue and not the one with the messages.

Ultimately, rebooting the server (I probably could have started/stopped the rabbitmq app and did a reset) and explicitly naming the queue that the output LS was to bind to fixed it.

One gotcha that I've found so far with rabbitmq is that @metadata info doesn't survive going through the queue. I had to add variables to my input plugins and save whatever @metadata I needed to those fields.