Are Logstash conf files combined in some way?

[pmarques (Paulo Marques)]

Hello All,

I might miss some information on the documentation...

I saved two different configuration files at /etc/logstash/conf.d which may run under 'main' pipeline

first.conf

input
  {
   file
     {
      path => "/data/first/*.csv"
      start_position => "beginning"
      sincedb_path => "/var/opt/sincedb/first.log"
     }
  }

filter {
  csv
    {
     separator => ";"
     columns => ["one","two","three"]
  mutate
    {
     remove_field => [ "message" ]
    }
}

output {
   elasticsearch
     {
      hosts => "http://localhost:9200"
      index => "first"
     }
}

second.conf

input
  {
   file
     {
      path => "/data/second/*.csv"
      start_position => "beginning"
      sincedb_path => "/var/opt/sincedb/second.log"
     }
  }

filter {
  csv
    {
     separator => ";"
     columns => ["one","three"]
  mutate
    {
     remove_field => [ "message" ]
    }
}

output {
   elasticsearch
     {
      hosts => "http://localhost:9200"
      index => "second"
     }
}

The issue is that when the first file is saved in /data/first, both indices (first and second) receive the data.

Are the file being combined in some way?

Thanks in advance,
Paulo

[Badger]

@pmarques

Are the file being combined in some way?

Yes. If -f points to a directory all the files in that directory are combined (including foo.conf, foo.conf-, foo.conf.bkup) into a single configuration. Events are read from all the inputs, put through all the filters, and sent to all the outputs.

If you want to have two pipelines then do not use -f and configure a pipelines.yml

[pmarques (Paulo Marques)]

Hello @Badger,

Thank you so much for your quick reply.

I do need different indices, so based on your answer, I will need different pipelines. I am testing this using the pipelines.yml file. Anything that I need to be aware of?

Best,
Paulo

[Badger]

No, it is pretty straightforward.

[pmarques (Paulo Marques)]

Hey @Badger,

Thanks for your help. It is working smoothly.

Paulo