Coexistence with logstash-forwarder

Elastic Stack Beats
1

I'm a bit confused about how FileBeat fits in with a logstash setup that already includes logstash-forwarder.

I have a couple of logstash config files for defining inputs for both LF and FB:

# 01-lumberjack-input.conf
input {
  lumberjack {
    port => 5000
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

and

# 02-beats-input.conf
input {
  beats {
    port => 5044
  }
}

In this test case, I am feeding IIS logs into it, so I have an IIS filter:

# 40-iis.conf
filter {
  if [type] == "iis" {
    # lots of stuff here
  }
}

I also have a general output config:

# 30-lumberjack-output.conf
output {
  elasticsearch {
    cluster => "ElasticCluster"
    host => localhost
    port => 9300
  }
}

What's confusing me is that the documentation puts in an output{} to the beats config, but I can't work out how I use that and not break the existing output I have already. How do I make Logstash use the @metadata-based output{} for beats stuff, but the normal output{} for everything else?

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
2

Use conditionals to select which output(s) to use. Presumably there is some field (perhaps [@metadata][beat] that's unique for Beats-generated events.

output {
  if [@metadata][beat] {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  } else {
    # non-Beats outputs
  }
}
3

Ah, okay that's easy then. That seems to be working, because I have a new index with that name. My next problem is the 'type' being set. If I hook up stdout { codec => rubydebug} I get two types defined, and I think my filter is looking at the other one.

A sample from my debug log:

{
    "message" => "2015-11-03 14:43:14 W3SVC12 ca-web04 10.117.1.14 GET /test.jpg - 80 - 10.117.1.22 useragent - domain 200 0 0 9369 452 29 1.2.3.4 http",
    "@version" => "1",
    "@timestamp" => "2015-11-03T14:44:26.376Z",
    "count" => 1,
    "fields" => {
        "type" => "iis"
    },
    "fileinfo" => {},
    "input_type" => "",
    "line" => 4,
    "offset" => 1763763,
    "shipper" => "MarkDesktop",
    "source" => "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex151103_x.log",
    "type" => "log"
}

So it looks like my filebeat.yml is just adding a field that happens to be called type to the data, rather than setting the actual type (which defaults to "log")

# filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - "C:/inetpub/logs/LogFiles/W3SVC1/*.log"
      fields:
        type: iis

output:
  elasticsearch:
    enabled: false

  logstash:
    enabled: true
    hosts: ["logstash:5044"]

shipper:

Could I perhaps be having a related problem as this post: Misleading type attribute in filebeat config? Maybe I need to wait for RC1.

4

are you using a nightly build? filebeat introduces 'document_type' in the prospectors config. The value in document_type will be used for 'type' in the published event.

5

I am using last night's build for the client, yes.

Blimey, that just worked :slight_smile: Thanks!