# Coexistence with logstash-forwarder

**URL:** https://discuss.elastic.co/t/coexistence-with-logstash-forwarder/33632
**Category:** Beats
**Tags:** filebeat
**Created:** [November 3, 2015, 1:22pm UTC](https://discuss.elastic.co/t/coexistence-with-logstash-forwarder/33632 "2015-11-03T13:22:18Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![Cylindric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cylindric/32/5660_2.png) [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [November 3, 2015, 1:22pm UTC](https://discuss.elastic.co/t/coexistence-with-logstash-forwarder/33632/1 "2015-11-03T13:22:18Z")

</div>

I'm a bit confused about how FileBeat fits in with a logstash setup that already includes logstash-forwarder.

I have a couple of logstash config files for defining inputs for both LF and FB:

```
# 01-lumberjack-input.conf
input {
  lumberjack {
    port => 5000
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

```

and

```
# 02-beats-input.conf
input {
  beats {
    port => 5044
  }
}

```

In this test case, I am feeding IIS logs into it, so I have an IIS filter:

```
# 40-iis.conf
filter {
  if [type] == "iis" {
    # lots of stuff here
  }
}

```

I also have a general output config:

```
# 30-lumberjack-output.conf
output {
  elasticsearch {
    cluster => "ElasticCluster"
    host => localhost
    port => 9300
  }
}

```

What's confusing me is that the documentation puts in an `output{}` to the beats config, but I can't work out how I use that and not break the existing output I have already. How do I make Logstash use the @metadata-based output{} for beats stuff, but the normal output{} for everything else?

```
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
```

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 3, 2015, 1:42pm UTC](https://discuss.elastic.co/t/coexistence-with-logstash-forwarder/33632/2 "2015-11-03T13:42:35Z")

</div>

Use conditionals to select which output(s) to use. Presumably there is some field (perhaps `[@metadata][beat]` that's unique for Beats-generated events.

```
output {
  if [@metadata][beat] {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  } else {
    # non-Beats outputs
  }
}
```

---

<div class="post-metadata">

### Author: ![Cylindric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cylindric/32/5660_2.png) [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [November 3, 2015, 2:54pm UTC](https://discuss.elastic.co/t/coexistence-with-logstash-forwarder/33632/3 "2015-11-03T14:54:51Z")

</div>

Ah, okay that's easy then. That seems to be working, because I have a new index with that name. My next problem is the 'type' being set. If I hook up `stdout { codec => rubydebug}` I get two types defined, and I think my filter is looking at the _other_ one.

A sample from my debug log:

```
{
    "message" => "2015-11-03 14:43:14 W3SVC12 ca-web04 10.117.1.14 GET /test.jpg - 80 - 10.117.1.22 useragent - domain 200 0 0 9369 452 29 1.2.3.4 http",
    "@version" => "1",
    "@timestamp" => "2015-11-03T14:44:26.376Z",
    "count" => 1,
    "fields" => {
        "type" => "iis"
    },
    "fileinfo" => {},
    "input_type" => "",
    "line" => 4,
    "offset" => 1763763,
    "shipper" => "MarkDesktop",
    "source" => "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex151103_x.log",
    "type" => "log"
}

```

So it looks like my filebeat.yml is just adding a field that happens to be called type to the data, rather than setting the actual type (which defaults to "log")

```
# filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - "C:/inetpub/logs/LogFiles/W3SVC1/*.log"
      fields:
        type: iis

output:
  elasticsearch:
    enabled: false

  logstash:
    enabled: true
    hosts: ["logstash:5044"]

shipper:

```

Could I perhaps be having a related problem as this post: [Misleading type attribute in filebeat config](https://discuss.elastic.co/t/misleading-type-attribute-in-filebeat-config/33221/3)? Maybe I need to wait for RC1.

---

<div class="post-metadata">

### Author: ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [November 3, 2015, 3:35pm UTC](https://discuss.elastic.co/t/coexistence-with-logstash-forwarder/33632/4 "2015-11-03T15:35:19Z")

</div>

are you using a nightly build? filebeat introduces 'document\_type' in the prospectors config. The value in document\_type will be used for 'type' in the published event.

---

<div class="post-metadata">

### Author: ![Cylindric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cylindric/32/5660_2.png) [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [November 3, 2015, 3:42pm UTC](https://discuss.elastic.co/t/coexistence-with-logstash-forwarder/33632/5 "2015-11-03T15:42:30Z")

</div>

I am using last night's build for the client, yes.

Blimey, that just worked 🙂 Thanks!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 9:58pm UTC](https://discuss.elastic.co/t/coexistence-with-logstash-forwarder/33632/6 "2017-07-05T21:58:27Z")

</div>


