# Filter out events by JSON content

**URL:** <https://discuss.elastic.co/t/filter-out-events-by-json-content/177037>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 16, 2019, 9:01am UTC](https://discuss.elastic.co/t/filter-out-events-by-json-content/177037 "2019-04-16T09:01:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![smoking81](https://avatars.discourse-cdn.com/v4/letter/s/d26b3c/32.png) [@smoking81](https://discuss.elastic.co/u/smoking81)\
**Post date:** [April 16, 2019, 9:01am UTC](https://discuss.elastic.co/t/filter-out-events-by-json-content/177037/1 "2019-04-16T09:01:40Z")

</div>

Hello there,  
I am running Filebeat on K8S and want to index just application files encoded in JSON which contain a field "classtype" with value "application". It seems a quite easy requirement, after trying all the possible combinations of json.message\_key, include\_lines and decode\_json\_fields I still didn't succeed in achieving what I wanted.. Can you please help me?

My extract from [https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml](https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml) looks now like this

```
  kubernetes.yml: |-
- type: docker
  containers.ids:
  - "*"
  # json.message_key: message
  # json.keys_under_root: true
  # json.overwrite_keys: true
  tail_files: true
  include_lines: ['"classtype":"application"']
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
    - decode_json_fields:
        fields: ["message"]
        target: ""
  # overwrite_keys: true

```

Thanks in advance!  
Regards

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [April 16, 2019, 4:18pm UTC](https://discuss.elastic.co/t/filter-out-events-by-json-content/177037/2 "2019-04-16T16:18:20Z")

</div>

What about using the `decode_json_fields` processor like you are, and then after that, using the `drop_event` processor with an appropriate `when` condition: [https://www.elastic.co/guide/en/beats/filebeat/current/drop-event.html](https://www.elastic.co/guide/en/beats/filebeat/current/drop-event.html)?

---

<div class="post-metadata">

**Author:** ![smoking81](https://avatars.discourse-cdn.com/v4/letter/s/d26b3c/32.png) [@smoking81](https://discuss.elastic.co/u/smoking81)\
**Post date:** [April 25, 2019, 10:16am UTC](https://discuss.elastic.co/t/filter-out-events-by-json-content/177037/3 "2019-04-25T10:16:03Z")

</div>

Hi @shaunak, thank you for replying! 🙂  
I think my problem is there are a lot of log entries from different sources and I want to do following:

- Discard non JSON
- Discard JSON not containing the field "classtype" with value of "application"

Example:

> {"log":"2019-04-24 14:52:35.571 I must be discarded\n","stream":"stdout","time":"2019-04-24T14:52:35.571865144Z"}  
> {"log":"{"severity":"INFO","classtype":"application","service":"my-service","trace":"","span":"","parent":"","exportab  
> le":"","pid":"11","thread":"scheduling-1","class":"MyJavaClass","message":"I must be fully indexed","stacktrace":""}\n","stream":"stdout","time":"2019-04-23T15:57:36.409648401Z"}  
> {"log":"{"severity":"INFO","service":"my-service","trace":"","span":"","parent":"","exportab  
> le":"","pid":"11","thread":"scheduling-1","class":"MyJavaClass","message":"I must NOT be indexed (no classtype)","stacktrace":""}\n","stream":"stdout","time":"2019-04-23T15:57:36.409648401Z"}

What happens using for example **json.message\_key: log** is that logs which are not JSON at all cause following exception:

> ERROR json/json.go:51 Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

Do you have any idea ho to achieve my goal? Thanks

---

<div class="post-metadata">

**Author:** ![smoking81](https://avatars.discourse-cdn.com/v4/letter/s/d26b3c/32.png) [@smoking81](https://discuss.elastic.co/u/smoking81)\
**Post date:** [April 30, 2019, 1:53pm UTC](https://discuss.elastic.co/t/filter-out-events-by-json-content/177037/4 "2019-04-30T13:53:20Z")

</div>

I am really stuck with this problem.. It seems a quite easy requirement but I don't understand how to solve it.  
Is the "\n" after the JSON which leads to the "Error decoding JSON" error for this log with both decode\_json\_fields fields ["log"] and json.message\_key: log?

```
"log":"{"severity":"INFO","classtype":"application","service":"my-service","trace":"","span":"","parent":"","exportab
le":"","pid":"11","thread":"scheduling-1","class":"MyJavaClass","message":"I must be fully indexed","stacktrace":""}\n"

```

Is there an easy solution for this? Thanks in advance!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 28, 2019, 1:53pm UTC](https://discuss.elastic.co/t/filter-out-events-by-json-content/177037/5 "2019-05-28T13:53:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.