Two indexes receiving same content regardless of logstash conf files

Elastic Stack Logstash
1

I've got two Logstash configuration files that each specify a different location for log files and then define the index for said log files.

The first:

input {
  file {
    path => "/elk/nmap/results/puppetserver/full/*"
    start_position => "beginning"
    sincedb_path => "/elk/nmap/sincedb/puppetserver"
  }
}

filter {
    json {
      source => "message"
    }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "puppetserver"
  }
}

Second:

input {
  file {
    path => "/elk/nmap/results/saltmaster/full/*"
    start_position => "beginning"
    sincedb_path => "/elk/nmap/sincedb/saltmaster"
  }
}

filter {
    json {
      source => "message"
    }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "saltmaster"
  }
}

For some reason, when I put logs into the puppetserver folder

/elk/nmap/results/puppetserver/full/

they get added to both indexes, and I can't figure out why. Any ideas?

Secondary question: nothing is being written to the sincedb files, so every time I restart logstash to add a new index, it re-adds all of the content again. File permissions are world write-able and read-able (I know those permissions are a bad idea, but I set them just for testing).

2

Config files are concatenation so data from all inputs will go to all outputs unless you use conditionals. It is a very common misunderstanding so you should be able to find many examples here.

3

Will it work with two different pipeline?

I had similar issue two config file was reading some data from database. some of the fields was same and output in index was getting messed up.

once I separated out both config under two different pipeline problem disappear.

4

Yes, explicitly defining them as separate pipelines works as well.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.