One Logstash file with multiple logs entries/pipelines

[Milad_Hamid]

Hi,

I have two different logs source ( Syslogs;logstash-syslog.conf, Apache logs;logstash-apache.conf), and i want to get the syslogs and apache logs into logstash and then ship them to elasticsearch with one logstash instance. So how i can implement this request instead of each time running this command to ship my logs to logstash:

bin/logstash -f path/config/logstash-apache.conf OR bin/logstash -f path/config/logstash-syslog.conf

[Christian_Dahlqvist]

If you are using Logstash 6.0, you can set it up to manage multiple pipelines in parallel based on your two configurations. In earlier versions you could place both files in a directory and point Logstash to this instead of the files. Logstash would then concatenate the files into a single pipeline (be sure you are using conditionals to separate the two flows).

[Milad_Hamid]

Thanks @Christian_Dahlqvist for your quick reply and help. The version that i use right now in my ELK stack is Logstash 5.6.4.

Since I am new to ELK world, please i want some basics steps how to implement the second approach realetd to old version of logstash to manage two config files. This is a screenshot of my Logstash/Config directory:

So what i have to edit and to do to get it run.

[Christian_Dahlqvist]

Put the config files in a separate directory and pass this to Logstash at startup. Make sure you have tagged the data in the filters and applied conditionals throughout before you do that though.

[Milad_Hamid]

Ok, so i have to put each config file in separate directory, then in logstash yaml file i have to add the paths of directories! which section in logstash settings file i have to edit to point to the directories? is it Pipeline Configuration Settings in logstash.yml file?

and regarding tags in filter part you meant this:
filter {
if [type] == "type-of-message-from-redis" {
...
}
if [type] == "type-of-message-from-file" {
...
}
}

output {
if [type] == "type-of-message-from-redis" {
elasticsearch {
host => "localhost"
}
}
if [type] == "type-of-message-from-file" {
elasticsearch {
host => "localhost"
index => "ssllogs-%{+YYYY.MM.dd}"
}
}
}

[Christian_Dahlqvist]

You put both files in the same directory (no other files) and then give that directory instead of a file name when you start Logstash.

[Milad_Hamid]

Ok, thanks i will test and post the result in case of success

[Milad_Hamid]

Yes thanks it is working in great way i have both logs now in logstash, and i can see data in Kibana.

So just to summarize because i know there are a lot people like me new to ELK, so I created a directory with name pipelines and i moved my logstash-syslog.conf and logstash-apache.conf files into this new directory, then i used following command to run logstash and ship my logs:
bin/logstash -f path/logstash-5.6.4/pipelines

Thank a lot @Christian_Dahlqvist