Logstash 6.2 Multiple Pipelines,each with its own output file

[davehatma (David Howell)]

I'm running three pipelines from 'pipelines.yml' each pipeline config file has its own output filter:

file {
    path => "/var/log/logstash/logstash-[my-pipeline name].log"
}

I'm finding that the output for all three configs is going to just one of the log files and appears to be the first one in pipelines.yml.

Is this expected behavior? If not is there a bug? If so, I think it would be a nice feature.

I like to keep the log files for each pipeline separate as they are easier to search and manage.

Thanks in advance,

Dave Howell

[Evesy (Michael Eves)]

Hi Dave,

Would you be able to share the contents of pipelines.yml & the respective configs that it references?

Providing your Logstash output is pointing to a different path in each pipeline I see no reason all your outputs would end up in the same file

[Suman_Reddy1 (Suman Reddy)]

Also, please paste your logstash startup commands

[davehatma (David Howell)]

Most certainly.

I'm using systemd to start /stop Logstash. Note that when I use the command line
instead (/usr/share/logstash/bin/logstash), I get the same behavior.

Here is the logstash.system file:

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

Here is the content of pipelines.yml with all commented lines removed:

- pipeline.id: metrics
  path.config: "/etc/logstash/conf.d/metrics.conf"
- pipeline.id: exceptions
  path.config: "/etc/logstash/conf.d/exceptions.conf"
- pipeline.id: transactions
  path.config: "/etc/logstash/conf.d/transactions.conf"

As I am unable to paste in the full content of the three config files (due to character limit for responding in the forum), here are the "output" filters for the three config files (which should be the only pertinent sections for the behavior I am experiencing):

metrics.conf:

output {
    elasticsearch {
        action => "index"
        hosts => "elasticlog-p01:9200"
        index => "%{app_group}::%{app_name}::metrics::%{[@metadata][index_env]}%{[@metadata][index_log_date]}"
    }

    file {
        path => "/var/log/logstash/logstash-metrics.log"
    }
}

exceptions.conf:

output {
    elasticsearch {
        action => "index"
        hosts => "elasticlog-p01:9200"
        index => "%{app_group}::%{app_name}::metrics::%{[@metadata][index_env]}%{[@metadata][index_log_date]}"
    }

    file {
        path => "/var/log/logstash/logstash-exceptions.log"
    }
}

transactions.conf:

output {
    elasticsearch {
        action => "index"
        hosts => "elasticlog-p01:9200"
        index => "%{app_group}::%{app_name}::transactions::%{[@metadata][index_env]}%{[@metadata][index_log_date]}"
    }
    
    file {
        path => "/var/log/logstash/logstash-transactions.log"
    }
}

[Badger]

Is your exceptions output really the same as the metrics output, or did you paste in the wrong config?

[davehatma (David Howell)]

No they are different. Paste error on my part. It should be '.../exceptions.log'. I have edited my reply to reflect. Thanks for pointing that out.

[davehatma (David Howell)]

Michael and Suman, with apologies please disregard this issue. I don't know how or why, but I came in this morning and it is working now. All three logs are being posted as desired.

Thanks for you help and sorry for the false alarm.