I'm learning ELK and had a setup for Topbeat (server metrics) and Filebeat going into a Logstash config over port 5000. That's working fine and I was able to load the Topbeat Dashboards, etc.
I wanted to use CollectD to monitor my Redis cluster (aside from the Topbeat metrics). So I created a new logstash conf (collectd.conf) and my config is below:
input {
udp {
port => 25826
buffer_size => 1452
codec => collectd { }
type => "collectd"
}
}
output {
elasticsearch {
hosts => ["xx.xx.xx.xx:9200"]
sniffing => true
manage_template => false
index => "collectd-%{+YYYY.MM.dd}"
document_type => "collectd"
}
}
I started getting CollectD metrics (I had recognized them from when I was testing out collectd before topbeat). My ELK was collecting data from 6 nodes and all 6 were sending data to the collectd-* indexes. None of these nodes were running collectd and once I stopped topbeat, it stopped collecting data. For reference, here is my beats.conf
input {
beats {
host => "xx.xx.xx.xx"
port => 5000
codec => "json"
}
}
output {
if [type] == "system" or [type] == "filesystem" or [type] == "process" {
elasticsearch {
hosts => ["xx.xx.xx.xx:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
if [type] == "nginx-access" {
elasticsearch {
hosts => ["xx.xx.xx.xx:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}
When I was using filebeat/topbeat, these were getting properly indexed. Once I configured the collectd.conf, I started receiving data already before collectd was running on my test node.