Are Logstash conf files combined in some way?

Elastic Stack Logstash
1

Hello All,

I might miss some information on the documentation...

I saved two different configuration files at /etc/logstash/conf.d which may run under 'main' pipeline

first.conf

input
  {
   file
     {
      path => "/data/first/*.csv"
      start_position => "beginning"
      sincedb_path => "/var/opt/sincedb/first.log"
     }
  }

filter {
  csv
    {
     separator => ";"
     columns => ["one","two","three"]
  mutate
    {
     remove_field => [ "message" ]
    }
}

output {
   elasticsearch
     {
      hosts => "http://localhost:9200"
      index => "first"
     }
}

second.conf

input
  {
   file
     {
      path => "/data/second/*.csv"
      start_position => "beginning"
      sincedb_path => "/var/opt/sincedb/second.log"
     }
  }

filter {
  csv
    {
     separator => ";"
     columns => ["one","three"]
  mutate
    {
     remove_field => [ "message" ]
    }
}

output {
   elasticsearch
     {
      hosts => "http://localhost:9200"
      index => "second"
     }
}

The issue is that when the first file is saved in /data/first, both indices (first and second) receive the data.

Are the file being combined in some way?

Thanks in advance,
Paulo

2

Yes. If -f points to a directory all the files in that directory are combined (including foo.conf, foo.conf-, foo.conf.bkup) into a single configuration. Events are read from all the inputs, put through all the filters, and sent to all the outputs.

If you want to have two pipelines then do not use -f and configure a pipelines.yml

3 Likes
3

Hello @Badger,

Thank you so much for your quick reply.

I do need different indices, so based on your answer, I will need different pipelines. I am testing this using the pipelines.yml file. Anything that I need to be aware of?

Best,
Paulo

4

No, it is pretty straightforward.

5

Hey @Badger,

Thanks for your help. It is working smoothly.

Paulo

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.