# Repeat: Filebeat include is not working when logs are in both json and non-json format

**URL:** <https://discuss.elastic.co/t/repeat-filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/134868>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 6, 2018, 8:24pm UTC](https://discuss.elastic.co/t/repeat-filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/134868 "2018-06-06T20:24:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cweiss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cweiss/32/30633_2.png) [@cweiss](https://discuss.elastic.co/u/cweiss)\
**Post date:** [June 6, 2018, 8:24pm UTC](https://discuss.elastic.co/t/repeat-filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/134868/1 "2018-06-06T20:24:51Z")

</div>

This is more or less a duplicate of the problem described here: [Filebeat include is not working when logs are in both json and non-json format](https://discuss.elastic.co/t/filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/126372?u=cweiss). It was closed without resolution ☹ .

We are trying to parse JSON content in live Jenkins build logs, we only want the JSON bits, not all the extra Jenkins log text.

prospector config:

```
- type: log
    paths: "/var/jenkins_home/jobs/*/jobs/*/jobs/test/builds/*/log"
    json.keys_under_root: true
    fields_under_root: true
    json.message_key: job_facts.BuildNumber
    include_lines:
      - '^{'

```

Test Jenkins output we're trying to parse:

```
Started by user Weiss, Chris
[Pipeline] node
Running on c5275fad5e1d-46ed17a9 in /var/swarm-client/workspace/Release_Engineering_Core/sandbox/test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Preparation)
[Pipeline] echo
{"job_facts.BuildNumber": "24"}
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

```

If we have the include\_lines statement, we get no output to Elasticsearch.

If we remove it, we get both the (correctly deconstructed) JSON but also the unwanted non-JSON log entries.

Either way, the filebeat log displays "Error decoding JSON" entries for each non-JSON line in the log.

I've also tried many variations on the "include\_lines" statement:

```
include_lines: ['^\{']
include_lines: '^{'

```

etc...

I should add that we're in control of the JSON structure of the content in the Jenkins logs, if the JSON is not formatted correctly for Filebeat, we can fix that.

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [June 7, 2018, 4:11am UTC](https://discuss.elastic.co/t/repeat-filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/134868/2 "2018-06-07T04:11:06Z")

</div>

Hi,

> [@](#):
>
> I've also tried many variations on the "include\_lines" statement:

Have you tried with below syntax. Please try it with and let me know if still not working.

> include\_lines: ["^ERR", "^WARN"]

Regards,

---

<div class="post-metadata">

**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\
**Post date:** [June 7, 2018, 5:45am UTC](https://discuss.elastic.co/t/repeat-filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/134868/3 "2018-06-07T05:45:07Z")

</div>

The problem here is that JSON decoding happens before filtering, so the `include_lines` is useless.

One way you can make it work is by adding a drop processor to get rid of the non-JSON events, like this:

```auto
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /tmp/json.log
  json.keys_under_root: true
  fields_under_root: true
  json.message_key: job_facts.BuildNumber

processors:
    - drop_event:
          when:
            regexp:
              message: ""

```

This will drop all events that contain a `message` field, that did the trick for me.

---

<div class="post-metadata">

**Author:** ![cweiss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cweiss/32/30633_2.png) [@cweiss](https://discuss.elastic.co/u/cweiss)\
**Post date:** [June 7, 2018, 3:08pm UTC](https://discuss.elastic.co/t/repeat-filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/134868/4 "2018-06-07T15:08:08Z")

</div>

Thanks, that worked for us as well!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2018, 3:08pm UTC](https://discuss.elastic.co/t/repeat-filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/134868/5 "2018-07-05T15:08:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.