# Omit non-json lines in log files while still allowing json parsing

**URL:** <https://discuss.elastic.co/t/omit-non-json-lines-in-log-files-while-still-allowing-json-parsing/214319>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 9, 2020, 12:29am UTC](https://discuss.elastic.co/t/omit-non-json-lines-in-log-files-while-still-allowing-json-parsing/214319 "2020-01-09T00:29:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gregory\_Zimmers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregory_zimmers/32/45582_2.png) [@Gregory\_Zimmers](https://discuss.elastic.co/u/Gregory_Zimmers)\
**Post date:** [January 9, 2020, 12:29am UTC](https://discuss.elastic.co/t/omit-non-json-lines-in-log-files-while-still-allowing-json-parsing/214319/1 "2020-01-09T00:29:08Z")

</div>

I've found that I'm only able to apply ['^{'] this pattern to include\_lines when I omit json parsing from the yml. After reading the documentation, it implies the line\_filtering is done after parsing, so if that's the case, how do I filter lines that I don't want included?

Examples below

This works

```auto
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /applogs/*.log*
    include_lines: ['^{']
# json.message_key: "level"
# json.keys_under_root: true
# json.overwrite_keys: true
processors:
  - add_fields:
      fields:
        kibanaspace: "specific-space"

```

This does not.

```auto
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /applogs/*.log*
    include_lines: ['^{']
    json.message_key: "level"
    json.keys_under_root: true
    json.overwrite_keys: true
processors:
  - add_fields:
      fields:
        kibanaspace: "specific-space"

```

given an input log file of

```auto
{"level": "INFO", "workerId": "5cced6bb522d-10-139641205327616", "traceId": null, "message": "Metrics blah", "datetime": "20-01-09 00:17:10:446539"}
{"level": "INFO", "workerId": "5cced6bb522d-10-139641205327616", "traceId": null, "message": "Metrics blah", "datetime": "20-01-09 00:17:10:446539"}
This is a 3rd party log that I dont want captured
Another unstructured log I want filtered out

```

---

<div class="post-metadata">

**Author:** ![faec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faec/32/46988_2.png) [@faec](https://discuss.elastic.co/u/faec)\
**Post date:** [January 9, 2020, 8:43pm UTC](https://discuss.elastic.co/t/omit-non-json-lines-in-log-files-while-still-allowing-json-parsing/214319/2 "2020-01-09T20:43:29Z")

</div>

Yes, line filtering happens after parsing. What happens in this case is that it tries to parse the incoming log line as json, then if it succeeds, it looks for a key `"level"` in the json and requires that it's a string beginning with `{` (which is not what you want).

Right now I don't believe there's a way to really interleave json and non-json, however if you only want to keep the json entries as in your example, then you should be able to just enable json without any filtering options, as lines that can't be decoded as json are ignored by default. In that case you probably just need:

```auto
  json.keys_under_root: true
  json.overwrite_keys: true

```

---

<div class="post-metadata">

**Author:** ![Gregory\_Zimmers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregory_zimmers/32/45582_2.png) [@Gregory\_Zimmers](https://discuss.elastic.co/u/Gregory_Zimmers)\
**Post date:** [January 10, 2020, 7:16pm UTC](https://discuss.elastic.co/t/omit-non-json-lines-in-log-files-while-still-allowing-json-parsing/214319/3 "2020-01-10T19:16:32Z")

</div>

For some reason, 99% of my third party logs that were not json were still being successfully parsed by filebeats and ingested by logstash/elastic. So instead, to filter out the lines that I want excluded post-filtering, I forced the existence of a field after parsing.

```auto
processors:
    - drop_event:
        when:
          not:
            has_fields: ["level"]

```

This worked in filtering out json parsed lines that did not contain the expected level field. I plan on adjusting my schema to conform to ECS 1.4. But this was a good first step in removing noise.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2020, 7:16pm UTC](https://discuss.elastic.co/t/omit-non-json-lines-in-log-files-while-still-allowing-json-parsing/214319/4 "2020-02-07T19:16:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.