Filter cross-pollination


(ethr bunny) #1

Im having issues with logstash ".conf" (modules?) and keeping their actions separated.

My logstash host is getting input from various beats and from collectd. Collectd sends its data via port 25826/UDP so my hope was to use this information to separate its values out. The beats are all default.

(This all stems from having the collectd index hitting the '1000 fields' error)

My collectd.conf:

input {
  udp {
    port => 25826
    buffer_size => 1452
    codec => collectd { }
  }
}
output {
        elasticsearch {
            hosts => [ "elasticsearch.myhost:9200" ]
            index => "collectd-%{+YYYY.MM.dd}"
        }
}

filter {
  if [beat][hostname] !~ /^(kafka|cass)/ {
    drop { }
  }  

  mutate {
    add_field => {
      "source" => "collectd"
    }
  }
}

Two things are happening:

1 - adding the 'drop' clause drops from both collectd AND beats
2- every value (collectd and beats) gets annotated with "source":"collectd"

Why?


(ethr bunny) #2

I think the issue here is pipeline.yml.

I updated it to look like this:

- pipeline.id: beats
  path.config: "/etc/logstash/conf.d/beats.conf"

- pipeline.id: collectd
  path.config: "/etc/logstash/conf.d/collectd.conf"
  pipeline.workers: 1

Now it seems to be working a bit better.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.