Trouble Running a Second Pipeline in Logstash 8.x (Multiple Pipelines Not Working)

Elastic Stack Logstash
1

Hello.
Please help me understand why the second pipeline in Logstash is not working. I have Logstash 8+ installed. I added a second pipeline path in the pipelines.yml file as follows:

- pipeline.id: audit
  path.config: "/etc/logstash/conf.d/audit_log/*.conf"
- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

Before that, I created the directory audit_log, provided the necessary permissions, and set the owner to logstash. I created a new pipeline test.conf which looks like this:

input {
  tcp {
    port => 1515
    type => "syslog"
  }
  udp {
    port => 2515
    type => "syslog"
  }
}

filter {
  if [message] =~ /\s+audit:/ {
    mutate {
      add_tag => ["audit"]
    }
    grok {
      match => {
        "message" => "<%{INT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}\s+%{HOSTNAME:hostname}\s+audit:\s+%{USER:command_user} \[%{INT:uid}\]:\s+%{INT:audit_pid}\s+%{TIMESTAMP_ISO8601:audit_timestamp}\s+%{DATA:command}\s+\[%{INT:result}\]\s+\(original user: %{USER:original_user}\)"
      }
    }
    ruby {
      code => "
        if event.get('result') == '0'
          event.set('result', 'Successfully')
        elsif event.get('result')
          event.set('result', 'Error. Return code: ' + event.get('result').to_s)
        end
      "
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://192.168.0.1:9200", "https://192.168.0.2:9200", "https://192.168.0.3:9200"]
    index => "audit-commands-%{+YYYY.MM.dd}"
    ssl => true
    ssl_certificate_verification => true
    cacert => "/etc/logstash/root.pem"
    user => "logstash"
    password => "xxx"
  }
}

This setup is not working. Could you please tell me what I am doing wrong?

2

Hello and welcome,

You need to share what you have in your logstash logs, without the logs is not possible to understand what could be the issue.

Also, what does your other logstash looks like? You only shared one pipeline configuration.

3

I found this information in the logs: "[2025-12-11T09:57:25,335][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified". This means the 'mail' pipeline is loading, but the second one is not. I don't understand why this is happening?

4

How did you start LS? As a process with -f /path/file.conf? That means it will run only a single file. Easiest way to start as a service.

When you start Logstash without arguments, it will read the pipelines.yml file and instantiate all pipelines specified in the file. On the other hand, when you use -e or -f , Logstash ignores the pipelines.yml file and logs a warning about it.

For multiple .conf files, you can use cmds:
logstash -f file1.conf -f file2.conf -f file3.conf

5

How are you starting logstash?

When using multiple pipelines you should run logstash as service with systemctl start logstash.

This may work, but is not the same as running multiple pipelines, this will merge all configurations into one, they will not be isolated and the inputs, filters and outputs will be applied to all events.

Using pipelines.yml with multiple pipelines isolate the pipelines from each other.

1 Like
6

Good point to underline. The pipeline isolate .conf files. However my point was that if you have multiple files in a directory, i.e. audit_log/.conf* than you can run as cmd by adding all files.