Multiple Logstash Configuration Files

[alpo]

All:

We're having an issue with Logstash and/or Elasticsearch, and we're not really sure where the issue resides. We are running Logstash and Elasticsearch on a single physical server and when viewing the data in Kibana, two indices incorrectly contain the same data. Our intent is to separate the data into two separate Elasticsearch indices, but we have so far been unsuccessful.

The local Logstash instance obtains log data from two separate files on the filesystem as configured by two separate Logstash configuration files located in /etc/logstash/conf.d. Each of the config files calls the elasticsearch plug-in using index => "logstash-indexname1" (in the first config file) and index => "logstash-indexname2" in the second config file. Of course in the "file" section, the required path variable is set accordingly and verified correct. The data are input to Elasticsearch, but unfortunately they are incorrectly merged into both indices when they should be separate.

As an aside, Logstash is started with systemd using ExecStart=/usr/share/logstash/bin/logstash --quiet -f /etc/logstash/conf.d --path.settings /etc/logstash

We wanted to ask the community what might cause this behavior.

Thanks in advance.

[Badger]

@alpo

-f /etc/logstash/conf.d

It sounds like you have two configuration file, each of which has an input, filters, and an output.

If you point path.config at a directory then it will concatenate all of the configuration files in that directory, read events from the inputs and send them to all of the outputs.

I would recommend configuring pipelines.yml, so that you use a different pipeline for each configuration file, or you could tag events on the inputs and use conditionals based on the tags.

[alpo]

Thanks, Badger. I'll give that a shot and report back.

[alpo]

Yup. That did it. I appreciate the help.

If anyone is having similar issues, check this out: https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html