# Condition with decode\_json\_fields processor
**URL:** https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093
**Category:** Beats
**Tags:** filebeat
**Created:** [January 11, 2018, 1:16pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093 "2018-01-11T13:16:53Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@jochenchrist](https://discuss.elastic.co/u/jochenchrist)
#### Post date: [January 11, 2018, 1:16pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/1 "2018-01-11T13:16:53Z")
Hi,
I try to collect docker logs with filebeats 6.1.
The application logs are written as JSON, which I want to decode with decode\_json\_fields processor.
Spring Boot's Bootstrapping also writes some plain log messages, so I need to decode\_json\_fields conditionally.
Example:
```auto
{"log":"\n","stream":"stdout","time":"2018-01-11T08:12:18.6298524Z"}
{"log":"\n","stream":"stdout","time":"2018-01-11T08:12:19.2633836Z"}
{"log":" . _______ _ _\n","stream":"stdout","time":"2018-01-11T08:12:19.2636263Z"}
{"log":" /\\\\ / ___'___ _ _(_)_ ____ _ \\ \\ \\ \\\n","stream":"stdout","time":"2018-01-11T08:12:19.2636359Z"}
{"log":"( ( )\\___ | '_ | '_| | '_ \\/ _` | \\ \\ \\ \\\n","stream":"stdout","time":"2018-01-11T08:12:19.2638339Z"}
{"log":" \\\\/ ___)| |_)| | | | | || (_| | ) ) ) )\n","stream":"stdout","time":"2018-01-11T08:12:19.2640029Z"}
{"log":" ' | ____| .__ |_| |_|_| |_\\__, | / / / /\n","stream":"stdout","time":"2018-01-11T08:12:19.2640115Z"}
{"log":" =========|_|==============|___/=/_/_/_/\n","stream":"stdout","time":"2018-01-11T08:12:19.2641995Z"}
{"log":" :: Spring Boot :: (v1.5.9.RELEASE)\n","stream":"stdout","time":"2018-01-11T08:12:19.273083Z"}
{"log":"\n","stream":"stdout","time":"2018-01-11T08:12:19.2731073Z"}
{"log":"{\"@timestamp\":\"2018-01-11T08:12:19.584+00:00\",\"@version\":1,\"message\":\"Starting DemoApplication v0.0.1-SNAPSHOT on caa07cb53010 with PID 1 (/app.jar started by root in /)\",\"logger_name\":\"com.example.demo.DemoApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000}\n","stream":"stdout","time":"2018-01-11T08:12:19.6096127Z"}
{"log":"{\"@timestamp\":\"2018-01-11T08:12:19.619+00:00\",\"@version\":1,\"message\":\"No active profile set, falling back to default profiles: default\",\"logger_name\":\"com.example.demo.DemoApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000}\n","stream":"stdout","time":"2018-01-11T08:12:19.6199775Z"}
```
This is my current config:
```
filebeat.prospectors:
- type: docker
paths:
- '/var/lib/docker/containers/*/*.log'
containers.ids: '*'
json.message_key: log
json.keys_under_root: true
json.overwrite_keys: true
processors:
- decode_json_fields:
when:
regexp:
log: "{\\\".*"
fields: ["log"]
target: ""
overwrite_keys: true
- add_docker_metadata: ~
output.elasticsearch:
hosts: ["elasticsearch:9200"]
```
I'd expect the regexp `when` condition to check if the log contains an encoded JSON, based on the documentation in [https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html](https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html), which states the `when` condition to be available for all processors.
However, in the filebeat log I get some errors:
```auto
2018/01/11 12:44:52.762759 json.go:32: ERR Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}
2018/01/11 12:44:52.762968 json.go:32: ERR Error decoding JSON: EOF
2018/01/11 12:44:52.763130 json.go:32: ERR Error decoding JSON: EOF
2018/01/11 12:44:52.763379 json.go:32: ERR Error decoding JSON: invalid character '.' looking for beginning of value
2018/01/11 12:44:52.763506 json.go:32: ERR Error decoding JSON: invalid character '/' looking for beginning of value
2018/01/11 12:44:52.763685 json.go:32: ERR Error decoding JSON: invalid character '(' looking for beginning of value
2018/01/11 12:44:52.763867 json.go:32: ERR Error decoding JSON: invalid character '\\' looking for beginning of value
2018/01/11 12:44:52.764031 json.go:32: ERR Error decoding JSON: invalid character '\'' looking for beginning of value
2018/01/11 12:44:52.764168 json.go:32: ERR Error decoding JSON: invalid character '=' looking for beginning of value
2018/01/11 12:44:52.764302 json.go:32: ERR Error decoding JSON: invalid character ':' looking for beginning of value
2018/01/11 12:44:52.764403 json.go:32: ERR Error decoding JSON: EOF
```
So, I am wondering, why filebeat tries to decode these entries at all.
Does `decode_json_fields` respect the when condition?
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [January 11, 2018, 2:51pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/2 "2018-01-11T14:51:36Z")
Try using single quotes around your regex so that you don't have to escape.
---
### Author:  [@jochenchrist](https://discuss.elastic.co/u/jochenchrist)
#### Post date: [January 11, 2018, 3:05pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/3 "2018-01-11T15:05:59Z")
Good advise, but no change with
```auto
processors:
- decode_json_fields:
when:
regexp:
log: '{\".*'
fields: ["log"]
target: ""
overwrite_keys: true
- add_docker_metadata: ~
```
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [January 11, 2018, 3:34pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/4 "2018-01-11T15:34:18Z")
Try disabling the decode\_json\_fields processor entirely. It seems to me that Filebeat is failing on the first JSON unmarshaling step despite the JSON appearing to be entirely valid.
---
### Author:  [@jochenchrist](https://discuss.elastic.co/u/jochenchrist)
#### Post date: [January 11, 2018, 4:22pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/5 "2018-01-11T16:22:17Z")
Well, the decode\_json\_fields processor actually works great for the vaild JSON log entries (the last 2 in the example above).
My concern are the error messages when a Non-JSON log entry is processed.
The question is, whether decode\_json\_fields respects the _when condition_, at all?
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [January 11, 2018, 4:37pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/6 "2018-01-11T16:37:23Z")
Oh, I didn't notice you were using `type: docker`. I believe that automatically sets up the JSON parsing for you (per the [docs](https://www.elastic.co/guide/en/beats/filebeat/6.1/configuration-filebeat-options.html#config-containers)). So I think you are actually running JSON parsing three times. So I believe these settings are redundant and should be removed.
```auto
json.message_key: log
json.keys_under_root: true
json.overwrite_keys: true
```
---
### Author:  [@jochenchrist](https://discuss.elastic.co/u/jochenchrist)
#### Post date: [January 12, 2018, 10:06pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/7 "2018-01-12T22:06:10Z")
Thanks Andrew for your replies.
I made some tests, and I think this simple config should work now:
```auto
filebeat.prospectors:
- type: docker
paths:
- '/var/lib/docker/containers/*/*.log'
containers.ids: '*'
processors:
- decode_json_fields:
fields: ["message"]
target: ""
overwrite_keys: true
- add_docker_metadata: ~
output.elasticsearch:
hosts: ["elasticsearch:9200"]
```
I'm still not fully sure about the original problem (probably the trick was `fields: ["message"]`), but this works now pretty stable, without the when condition and without any errors produced by filebeats.
---
### Author:  [@Asher\_Shoshan](https://discuss.elastic.co/u/Asher_Shoshan)
#### Post date: [January 14, 2018, 9:33am UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/8 "2018-01-14T09:33:28Z")
Hi,
Have you accomplished what you wanted? i.e the "message" field is parsed, and you can see the different fields in Kibana?
---
### Author:  [@jochenchrist](https://discuss.elastic.co/u/jochenchrist)
#### Post date: [January 14, 2018, 1:21pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/9 "2018-01-14T13:21:31Z")
Hi!
Well, I have a setup / workaround that works for my use case without error entries in the filebeat log.
However, the actual question of this topic is still unanswered:
> Does the `decode_json_fields` processor respect the _when_ condition?
In my experiments it did not.
But I'd expect it does, based on the documentation in [Define processors | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html)
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [January 16, 2018, 6:18pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/10 "2018-01-16T18:18:46Z")
> [@jochenchrist](#):
>
> Does the decode\_json\_fields processor respect the when condition?
Yes, it does. The conditions are implemented outside of the processors. The individual processors don't get a "say" in whether or not they respect the conditions.
Here's a simple example.
```auto
$ cat input.json
{"log": "hello world!"}
{"log": "hello world!", "counter": 42}
```
```auto
$ cat filebeat.json.yml
filebeat.prospectors:
- paths:
- 'input.json'
processors:
- decode_json_fields:
when.regexp.message: '^{'
fields: ["message"]
target: ""
overwrite_keys: true
output.file:
path: 'out'
filename: filebeat.json
```
```auto
$ cat out/filebeat.json | jq .
{
"@timestamp": "2018-01-16T18:16:15.520Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.1.1"
},
"source": "/Users/akroh/Downloads/filebeat-6.1.1-darwin-x86_64/input.json",
"offset": 24,
"message": "{\"log\": \"hello world!\"}",
"beat": {
"version": "6.1.1",
"name": "x",
"hostname": "x"
},
"log": "hello world!"
}
{
"@timestamp": "2018-01-16T18:16:15.521Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.1.1"
},
"beat": {
"name": "x",
"hostname": "x",
"version": "6.1.1"
},
"log": "hello world!",
"counter": 42,
"source": "/Users/akroh/Downloads/filebeat-6.1.1-darwin-x86_64/input.json",
"offset": 63,
"message": "{\"log\": \"hello world!\", \"counter\": 42}"
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 13, 2018, 6:18pm UTC](https://discuss.elastic.co/t/condition-with-decode-json-fields-processor/115093/11 "2018-02-13T18:18:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.