Pipeline startup/shutdown ordering when using multiple pipeline

Elastic Stack Logstash
1

Hi,

I am using Logstash-6.1.1 and its multiple pipeline feature. I need to send data to elasticsearch and hadoop with the caveat that hadoop service may become unavailable (it is managed by a second team and is located in a different location). Thus I need events to keep going to elasticsearch even if hadoop goes down.

To solve this, I have structured my logstash architecture to use two pipelines.

  • main - This pipeline reads the inputs and processes them and writes the events to elasticsearch and sends them over tcp to second pipeline (host - 127.0.0.1 and port - 12345).
  • hadoop - Input is tcp (port 12345, events sent by first pipeline). Output is the hadoop service.

The architecture works fine but I have noticed that hadoop pipeline is always started/stopped first (possibly because it has so few filters). This leads to a shutdown stall as during shutdown hadoop terminates first and stops receiving packets on port 12345. Thus main pipeline is stuck waiting for events to be sent over TCP. Shutting down logstash shows the following message

Pipeline terminated {"pipeline.id"=>"hadoop"}

followed by a bunch of "inflight_count"=> messages where the count does not vary. The shutdown eventually completes but I NEVER see a message like

Pipeline terminated {"pipeline.id"=>"main"}

Is it possible to specify pipeline ordering when using multiple pipelines or is there a hack (like giving them alphabetic names) which will ensure that one pipeline is started/stopped before the other?

2

To answer my own question:

It appears the pipelines are started in alphabetic order. Giving them names like "0_main" and "1_hadoop" ensures that main is always started and stopped before hadoop.

I couldn't find this in the documentation of the multiple pipelines.

3

I couldn't find this in the documentation of the multiple pipelines.

That's an indication that you shouldn't rely on that behavior.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.