Multiple Beats Input (filebeat, topbeat)

Elastic Stack Logstash
1

So I've configured an ELK stack. Initially, I configured my logging to use json format and used filebeat to send to logstash. Once I got that working, I wanted to start collecting system metrics so I disabled the filebeat configuration and created a topbeat one.

I'm not curious, how can I combine these two using a single beats input in logstash? I've been reading up on conditionals and was thinking of setting conditionals on document_type but I don't see that option in topbeat.

Ideally, I would like to configure a single port to be used and not have multiple logstash configs (with multiple ports open).

2

I'm wondering if setting a conditional on type would be advised?

Configuration

input {
  beats {
    host => "x.x.x.x"
    port => 5000
  }
}
output {
  if [type] == "system" or [type] == "filesystem" or [type] == "process" {
    elasticsearch {
      hosts => ["x.x.x.x:9200"]
      sniffing => true
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
      template_name => "topbeat"
      template => "/etc/logstash/mappings/topbeat.json"
    }
  }
  if [document_type] == "nginx-access" {
    do something here
  }
}

system, filesystem and process were the only fields I could see would be unique in conjunction with whatever my application logs would be sending. I'm setting the document_type in Filebeat but I don't see this option in Topbeat.

3

I think I solved my question, here's the config I'm using for reference:

input {
  beats {
    host => "x.x.x.x"
    port => 5000
    codec => "json"
  }
}
output {
  if [type] == "system" or [type] == "filesystem" or [type] == "process" {
    elasticsearch {
      hosts => ["x.x.x.x:9200"]
      sniffing => true
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
      template_name => "topbeat"
      template => "/etc/logstash/mappings/topbeat.json"
    }
  }
  if [type] == "nginx-access" {
    elasticsearch {
      hosts => ["x.x.x.x:9200"]
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    }
  }
}
2 Likes