Logstash pipeline shutting off/starting order

Elastic Stack Logstash
1

How to force logstash to switch off pipelines in given order?

In my setup, I’m often receiving data via http input and distributing those messages among various pipelines based on content, however if I turn the logstash off, pipelines with elasticsearch output are turned off first so data in pipeline with http input just got stuck (and the logstash has to be “kill -9”ed.

DLQ didn’t help, it wont save stuck messages, and only error I get is “Maybe the destination pipeline is down or stopping? Will Retry.”

2

Are you using pipeline-to-pipeline links? If so, I don’t think the ordering is documented, but at least one bug where shutdown hung due to ordering has occurred and been fixed. I believe that fix is specific to pipeline-to-pipeline and would not apply if you were using (for example) tcp-to-tcp.

Which version are you running and what does your set of inputs and outputs look like?

3

Yes, it is simple pipe-to-pipe configuration:

input {
      http {
        port => 8081
        ecs_compatibility => "disabled"
      }
}
filter {
...
}
output {
        pipeline { send_to => "elasticpipeline" }
}

In logs, I can clearly see the proper termination of the “elasticpipeline”, however, since there are few more pipes to close, the “httppipeline” is closed (almost) last, so after a while i just get deadlock and message “Maybe the destination pipeline is down…”.

The bug you mentioned may be related to “beats” input maybe? That is quite stable, didn’t find any issues (deadlocks) with that so far.

4

I don’t think there is anything related to different inputs in the way thing works.

When the pipeline output starts, it registers its send_to value, allowing the pipeline bus to know which outputs write to each input.

When the the shutdown method of the agent is called, it calls pipeline_bus.setBlockOnUnlisten, which sets blockOnUnlisten true (it defaults to false). When the pipeline input is asked to shutdown, it will block if there are any pipeline outputs that send data to it (the same is true in the old V1 pipeline bus). I’m not really in a position to dig any deeper in the PR that implemented the V2 bus.

So the elasticpipeline input should be in a delay loop waiting for the elasticpipeline output to shutdown.

Does elasticpipeline actually complete its shutdown, or does it just finish shutting down its outputs?

There was a time when pipelines were shutdown in alphabetical order so you could help it by naming them 01_foo, 02_bar, etc. I have no idea if that is still true, and don’t know if it would fix anything (it may just shorten the time window for the problem to occur in).

1 Like