# Configuring filebeat logstash plugin to transform one log, but not another

**URL:** <https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945>\
**Category:** Logstash\
**Created:** [December 14, 2018, 4:53pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945 "2018-12-14T16:53:47Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![swright-eti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swright-eti/32/38743_2.png) [@swright-eti](https://discuss.elastic.co/u/swright-eti)\
**Post date:** [December 14, 2018, 4:53pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945/1 "2018-12-14T16:53:47Z")

</div>

I have hosts that have log files in different formats, Vision, and Triad. They are being collected by Filebeat and shipped to Logstash. I have a Logstash config file setup for Beats as shown in the docs. At this point its pretty much a pass through (See below). I want to be able to transform the Triad logs, but leave the Vision logs alone.

So I am a little confused about how tell Logstash to differentiate between the two types of logs that it will be seeing. Its not clear to me from the Beats input plugin [docs](https://www.elastic.co/guide/en/logstash/6.5/plugins-inputs-beats.html) on how to do that in the input section.

```
input {
  beats {
    port => 5044
  }
}

filter {
} 

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
 }
}
```

---

<div class="post-metadata">

**Author:** ![swright-eti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swright-eti/32/38743_2.png) [@swright-eti](https://discuss.elastic.co/u/swright-eti)\
**Post date:** [December 14, 2018, 5:21pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945/2 "2018-12-14T17:21:51Z")

</div>

This is my Filebeat configuration for the inputs

```
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log
  enabled: true
  paths:
     - /usr/cbridge/msg/LaunchPad/*.log
     - /usr/cbridge/msg/Manager/*.log
     - /usr/cbridge/msg/Reporting/*.log
     - /usr/cbridge/msg/WorkOrderPrinting/*.log
  # Handle Java exceptions and put them on one line
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'  
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true
  paths:
     - /usr/cbridge/msg/current.msg
  # Exlude the line of dashes TODO this seems to get ignored when multiline is working. Probably need to strip it in logstash
  exclude_lines: ['^-+$']
  # Setup the pattern to harvest the multiline
  multiline.pattern: '^[A-Z]+: '
  multiline.negate: true
  multiline.match: after filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log
  enabled: true
  paths:
     - /usr/cbridge/msg/LaunchPad/*.log
     - /usr/cbridge/msg/Manager/*.log
     - /usr/cbridge/msg/Reporting/*.log
     - /usr/cbridge/msg/WorkOrderPrinting/*.log
  # Handle Java exceptions and put them on one line
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'  
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true
  paths:
     - /usr/cbridge/msg/current.msg
  # Exlude the line of dashes TODO this seems to get ignored when multiline is working. Probably need to strip it in logstash
  exclude_lines: ['^-+$']
  # Setup the pattern to harvest the multiline
  multiline.pattern: '^[A-Z]+: '
  multiline.negate: true
  multiline.match: after
```

---

<div class="post-metadata">

**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\
**Post date:** [December 14, 2018, 7:32pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945/3 "2018-12-14T19:32:53Z")

</div>

I'm pretty new to Elastic so I could be way off base here but what you might be looking to do is filter the data. I'd take a look at, [https://www.elastic.co/guide/en/logstash/6.5/config-examples.html](https://www.elastic.co/guide/en/logstash/6.5/config-examples.html) and [https://www.elastic.co/guide/en/logstash/6.5/filter-plugins.html](https://www.elastic.co/guide/en/logstash/6.5/filter-plugins.html). Again the experts know a lot more about this than me but hopefully I'm pointing you in the right direction. Sorry if this doesn't help.

This may be a useful link as well, [https://www.elastic.co/guide/en/logstash/6.5/transformation.html](https://www.elastic.co/guide/en/logstash/6.5/transformation.html).

---

<div class="post-metadata">

**Author:** ![swright-eti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swright-eti/32/38743_2.png) [@swright-eti](https://discuss.elastic.co/u/swright-eti)\
**Post date:** [December 14, 2018, 8:12pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945/4 "2018-12-14T20:12:07Z")

</div>

Thanks for the response Ryan, I appreciate it. I think the basic issue is that I have one pipeline definition "vision-logs-pipeline.conf". It has one input section to listen for Beats. The Beat sends both log types to Logstash, and they appear at the input section. Once I have filtered I dont know how to send the filtered data to the correct output.

Vision logs = A  
Triad logs = B

```
Filebeat(A, B) -> LogstashInput() -> Filter( A -> output1, B-> output2) -> Output1(A) -> ElasticIndex1
                                                                        -> Output2(B) -> ElasticIndex2

```

Or something like that anyway.

---

<div class="post-metadata">

**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\
**Post date:** [December 14, 2018, 8:20pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945/5 "2018-12-14T20:20:54Z")

</div>

I may be misunderstanding you but is this what you're looking to do:

> [@Can I filter the Output data separately?!](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064):
>
> We have the following env: LS 2.0.x; ES 2.0.x and K 4.2.x The input comes from the trace logs that an application produces, we filter the data and then wish to populate two separate (but related) indices: orders and orders\_alt. Each one of these has a separate mapping structure of data. We are having trouble controlling what fields go with what Index. We end up with the same fields in both Indices because ES creates them dynamically and we have to way to filter between the Outputs. How c…

---

<div class="post-metadata">

**Author:** ![swright-eti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swright-eti/32/38743_2.png) [@swright-eti](https://discuss.elastic.co/u/swright-eti)\
**Post date:** [December 14, 2018, 8:40pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945/6 "2018-12-14T20:40:53Z")

</div>

This is the solution I came up with. It appears to be working. I have no clue whether its efficient or not.

```
input {
  beats {
    port => 5044
  }
}

output {
  if [source] == '/usr/cbridge/msg/current.msg' {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "triad-%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\
**Post date:** [December 14, 2018, 8:46pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945/7 "2018-12-14T20:46:17Z")

</div>

Glad to hear its working, hopefully someone a little more advanced than me can help you streamline it if it even needs to be. Your many levels above where I'm at, good for you!!!

---

<div class="post-metadata">

**Author:** ![swright-eti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swright-eti/32/38743_2.png) [@swright-eti](https://discuss.elastic.co/u/swright-eti)\
**Post date:** [December 14, 2018, 8:59pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945/8 "2018-12-14T20:59:10Z")

</div>

Haha, well after all that, I sort of realized that I probably dont want to split it into separate indexes. Ah well at least I learned something interesting.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2019, 9:01pm UTC](https://discuss.elastic.co/t/configuring-filebeat-logstash-plugin-to-transform-one-log-but-not-another/160945/9 "2019-01-11T21:01:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.