Logstash pipeline creation is behaving differently for different outputs

Elastic Stack Logstash
1

We have two deployments where lumberjack is configured as output for one of the deployment and syslog is configured as output for another deployment.

In both the deployments the outputs are down when Logstash is initiated, following is the behavior observed.

  1. With Syslog as output, we could see Pipeline creation has been instantiated and the downstream queue started to fill.
  2. With lumberjack as output, Pipeline creation has not been instantiated, and the upstream queue started to fill before downstream.

When we observe the corresponding plugin code.
For syslog1 when there is a connection failure, a exception is raised rather performing indefinite retries unlike for other output like lumberjack2

Could someone provide some insights on why there is a difference b/n syslog and lumberjack.

2

We build a logstash-output-lumberjack by commenting the retry 1 and then performed following test.
Instantiated logstash without having lumberjack output available, we could see downstream pipeline i.e., lumberjack pipeline starting to fill.
The same test has been tried with vanila logstash-output-lumberjack plugin and we noticed "Pipeline has not created" and the upstream pipeline started to fill but downstream queue is empty.

3

Hi,

Can you share the logstash starting logs suggesting there might be an issue with the pipeline ? also can you share lumberjack output configurations ?

4

Hi @grumo35 ,

We have total 3 outputs configured, in the reference deployment

  1. Search Engine
  2. Lumberjack 1
  3. Lumberjact 2
    As part of this test Lumberjack 2 is not available during instantiating of logstash process.

Please find the logs and configurations that are requested

  1. Lumberjack 2 output configuration
  2. Logstash logs