Logstash With RabbitMQ Never Finish Process

Elastic Stack Logstash
1

Hi everyone, I've a problem when i'm running my logstash config with the rabbitmq input plugin, the process never ends, it keeps running forever. I would like to finish logstash process when all my queue documents was processed.

My config:

input {
  rabbitmq {
        host => "myhost"
        queue => "fnc_calculo"
        durable => true
        password => "guest"
        user => "guest"
    }
}
output {
  elasticsearch {
    codec => rubydebug
    hosts => "elastic:9200"
    index => "tb_fnc_calculo"
      document_type => "fnc_calculo"
  }
  stdout { codec => rubydebug }
}

*I'm using logstash 6.1.2

2

That's not supported by the rabbitmq input.

3

Right, although, how i can work with a CI tool like jenkins? Because if i keep it running, it doesn't get new docs uploaded to the queue, and if a close process by jenkins timeout configuration, it doesn't finish as sucess but as warning.

Thanks for you attention

4

Right, although, how i can work with a CI tool like jenkins?

Running Logstash with a rabbitmq input in a Jenkins job doesn't strike me as a very good idea. Jenkins is meant for time-bound jobs that are triggered manually, via cron, or via some other external trigger. Continuously fetching messages from a queue and indexing them in ES doesn't fit doesn't follow either of those patterns.

Because if i keep it running, it doesn't get new docs uploaded to the queue

What do you mean? If Logstash is running with a rabbitmq input it'll get all message arriving to the queue until it's shut down.

5

Running Logstash with a rabbitmq input in a Jenkins job doesn't strike me as a very good idea. Jenkins is meant for time-bound jobs that are triggered manually, via cron, or via some other external trigger. Continuously fetching messages from a queue and indexing them in ES doesn't fit doesn't follow either of those patterns.

@magnusbaeck I was wrong about rabbitmq concept. All my logstashs are running in a jenkins and i was trying put the rabbitmq integration as well.

It's clear now, how it works and how i should develop it. Thanks for your time and answer.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.