# Running filebeat modules alongside a custom pipeline

**URL:** <https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272>\
**Category:** Logstash\
**Created:** [December 10, 2020, 12:22pm UTC](https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272 "2020-12-10T12:22:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![pbatley](https://avatars.discourse-cdn.com/v4/letter/p/a6a055/32.png) [@pbatley](https://discuss.elastic.co/u/pbatley)\
**Post date:** [December 10, 2020, 12:22pm UTC](https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272/1 "2020-12-10T12:22:52Z")

</div>

I am completely new to Elastic so apologies if this is trivial.

I have managed to set up the ELK stack (self managed, all on same host machine) and have got filebeat's apache module enable sending logs to logstash. I use the following pipeline as per the examples to parse them:

```auto
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => ["http://172.17.17.50:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}" 
    }
  } else {
    elasticsearch {
      hosts => ["http://172.17.17.50:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

```

I now want to use filebeat to read apache style logs from a node/express application. I have set up an additional beat in filebeat.yml with basic settings:

```auto
- type: log
  enabled: true
  paths: 
    - /var/log/supervisor/express-api*.log

```

This seems to work as the log lines are making it to kibana unfiltered. (i.e. the message field contains the entire log line)

As I understand it, I can only have a single pipeline listening on a single port. So the point I am stuck at is: how do I set up logstash to filter these logs whilst still having the aforementioned apache pipeline also running?

Thanks in advance,

P

---

<div class="post-metadata">

**Author:** ![Raynald](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raynald/32/78897_2.png) [@Raynald](https://discuss.elastic.co/u/Raynald)\
**Post date:** [December 10, 2020, 1:01pm UTC](https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272/2 "2020-12-10T13:01:08Z")

</div>

You can add field in your filebeat.yml file, and then add filtering in your pipeline.

Example :  
`processors:`  
` - add_tags:`  
` tags: [PRODUCTION]`  
` target: "environment"`

Is this your question ?

---

<div class="post-metadata">

**Author:** ![pbatley](https://avatars.discourse-cdn.com/v4/letter/p/a6a055/32.png) [@pbatley](https://discuss.elastic.co/u/pbatley)\
**Post date:** [December 10, 2020, 1:06pm UTC](https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272/3 "2020-12-10T13:06:13Z")

</div>

Hi Raynald, thanks for your reply.

I had seen that one could add fields and tags to the beat but am still none the wiser as to how to then apply filters to only records that contain those fields/tags. My understanding is that every record will be parsed by the pipeline listening on 5044 though so any filters I write will also be applied to filebeat inputs that don't need them?

---

<div class="post-metadata">

**Author:** ![Raynald](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raynald/32/78897_2.png) [@Raynald](https://discuss.elastic.co/u/Raynald)\
**Post date:** [December 10, 2020, 1:11pm UTC](https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272/4 "2020-12-10T13:11:19Z")

</div>

You can filter in your pipeline for a specific value in field, like this :

`input {`  
`...`  
`}`

`filter {`  
` if [environment] == "PRODUCTION" {`  
` ...`  
` }`

`output {`  
` if [environment] == "PRODUCTION" {`  
` ...`  
` }`  
`}`

---

<div class="post-metadata">

**Author:** ![pbatley](https://avatars.discourse-cdn.com/v4/letter/p/a6a055/32.png) [@pbatley](https://discuss.elastic.co/u/pbatley)\
**Post date:** [December 10, 2020, 3:16pm UTC](https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272/5 "2020-12-10T15:16:39Z")

</div>

That's the ticket! Thank you very much.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 7, 2021, 3:16pm UTC](https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272/6 "2021-01-07T15:16:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.