# Include\_lines in My Filebeat Module

**URL:** <https://discuss.elastic.co/t/include-lines-in-my-filebeat-module/149483>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 21, 2018, 3:24pm UTC](https://discuss.elastic.co/t/include-lines-in-my-filebeat-module/149483 "2018-09-21T15:24:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![stevetso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stevetso/32/27438_2.png) [@stevetso](https://discuss.elastic.co/u/stevetso)\
**Post date:** [September 21, 2018, 3:24pm UTC](https://discuss.elastic.co/t/include-lines-in-my-filebeat-module/149483/1 "2018-09-21T15:24:09Z")

</div>

I have created my module in filebeat and my log can be ingested.

For filebeat.input, there is a feature called "include\_lines", which we could only include the lines which matched the regex.

In filebeat module, I tried to add "include\_lines" in modules.d/mymodule.yml but the filtering is not working.

If I use filebeat.input, the fields cannot be parsed correct because the log is in my customised format.

May I ask how could I use include\_lines in my own filebeat module? Thanks.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [September 25, 2018, 5:59am UTC](https://discuss.elastic.co/t/include-lines-in-my-filebeat-module/149483/2 "2018-09-25T05:59:02Z")

</div>

Can you share the exact config you were using? Have a look here on how variables are configured in modules: [https://www.elastic.co/guide/en/beats/filebeat/current/specify-variable-settings.html](https://www.elastic.co/guide/en/beats/filebeat/current/specify-variable-settings.html)

---

<div class="post-metadata">

**Author:** ![stevetso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stevetso/32/27438_2.png) [@stevetso](https://discuss.elastic.co/u/stevetso)\
**Post date:** [September 25, 2018, 6:31am UTC](https://discuss.elastic.co/t/include-lines-in-my-filebeat-module/149483/3 "2018-09-25T06:31:53Z")

</div>

I created my own module (named as "log4j") with based on the [Creating a New Filebeat Module | Beats Developer Guide [master] | Elastic](https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html)

After that I did the followings:

1. Copied the module into /usr/share/filebeat/module/
2. Copied the config file log4j.yml into /etc/filebeat/modules.d/

Here is the content of the module config in /etc/filebeat/modules.d/:

```
- module: log4j
  applogs:
    enabled: true
     var.paths: ["/path/to/log/*log"]
     include_lines: ["SUCCESS"]

```

I have the following questions:

1. It is my subjective thinking that "include\_lines" is working in modules. (i.e. apache2, Nginx or custom modules).
2. In official documentation website, "include\_lines" is not mentioned within the module section. Instead, "include\_lines" is covered in the filebeat.intput section ([Log input | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html))

I doubt how "include\_lines" (i.e. filter the logs before ingest into ES) works when using modules because we do have scenario that filter the logs and ingest what we want only.

I have a workaround now:

1. Use filebeat.input to configure the filebeat to read the log files, and use the "include\_lines".

2. When ingest into ES, I specify the pipeline used.

> output.elasticsearch:
> 
> hosts: ["elk:8080"]  
> protocol: "http"  
> username: "elasticsearch"  
> password: "password"  
> index: "logs-jboss-%{+yyyy.MM.dd}"  
> pipeline: filebeat-6.4.0-log4j-applogs-default

This workaround is working fine but all the input will be processed with same pipeline!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 23, 2018, 6:31am UTC](https://discuss.elastic.co/t/include-lines-in-my-filebeat-module/149483/4 "2018-10-23T06:31:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.