# Unable to parse docker json-file **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 19, 2017, 11:19am UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416 "2017-12-19T11:19:58Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Asaf\_Shabat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asaf_shabat/32/24442_2.png) [@Asaf\_Shabat](https://discuss.elastic.co/u/Asaf_Shabat)\ **Post date:** [December 19, 2017, 11:19am UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416/1 "2017-12-19T11:19:58Z")
Hello, I'm trying to parse docker json-file output to Logstash using Filebeat and to break the message log into fields. Currently, this is what I can parse into Kibana: ``` { "_index": "filebeat-2017.12.19", "_type": "doc", "_id": "pr50bmABIujUTb2Ryv8M", "_version": 1, "_score": null, "_source": { "offset": 6761353, "log": "\u001b[0m\u001b[0m11:07:12.951 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 54) WFLYUT0021: Registered web context: /rest\n", "prospector": { "type": "log" }, "source": "/var/lib/docker/containers/7688c378a4de513a8c5e587843512476ea996700e93a792e71c5962a190bb779/7688c378a4de513a8c5e587843512476ea996700e93a792e71c5962a190bb779-json.log", "message": "{\"log\":\"\\u001b[0m\\u001b[0m11:07:12.951 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 54) WFLYUT0021: Registered web context: /rest\\n\",\"stream\":\"stdout\",\"time\":\"2017-12-19T11:07:12.952311089Z\"}", "docker": { "container": { "name": "wildfly01", "image": "dockerepos.dom.local:5000/wildfly:10.0.0.Final", "id": "7688c378a4de513a8c5e587843512476ea996700e93a792e71c5962a190bb779", "labels": { "license": "GPLv2", "build-date": "20170801", "vendor": "CentOS" } } }, "tags": [ "beats_input_codec_plain_applied", "_grokparsefailure", "_jsonparsefailure" ], "@message": { "time": "2017-12-19T11:07:12.952311089Z", "log": "\u001b[0m\u001b[0m11:07:12.951 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 54) WFLYUT0021: Registered web context: /rest\n", "stream": "stdout" }, "@timestamp": "2017-12-19T11:07:14.787Z", "stream": "stdout", "@version": "1", "beat": { "name": "server_devenv01", "hostname": "server_devenv01", "version": "6.0.1" }, "host": "server_devenv01", "topic": "Local-Dev-wildfly", "time": "2017-12-19T11:07:12.952311089Z" }, "fields": { "@message.time": [ "2017-12-19T11:07:12.952Z" ], "@timestamp": [ "2017-12-19T11:07:14.787Z" ] }, "sort": [ 1513681634787 ] } ``` I want to break the @message field into some other fields, by the following pattern: %{DATE:date} %{TIME:time} %{LOGLEVEL:loglevel}%{SPACE} [(?[^]]+)] ((?[^]]+))%{SPACE} %{GREEDYDATA:message} Currently I have the following configurations: **filebeat.yml:** filebeat.prospectors: ``` - type: log enabled: true paths: - '/var/lib/docker/containers/*/*.log' processors: - add_docker_metadata: ~ fields: topic: Local-Dev-wildfly fields_under_root: true multiline.pattern: '^\[[:space:]]+|]$' multiline.match: after ``` **/etc/logstash/conf.d/10-filter.conf** > filter { > json { > source =\> "message" > target =\> "@message" > } > json { > source =\> "message" > } > grok { > match =\> { '@message' =\> '%{DATE:date} %{TIME:time} %{LOGLEVEL:loglevel}%{SPACE} [(?[^]]+)] ((?[^]]+))%{SPACE} %{GREEDYDATA:message}' } > } > } How can I break the @message log to a specific fields (LOGLEVEL, LOGGER, THREAD and etc...? Thanks! ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [December 19, 2017, 12:01pm UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416/2 "2017-12-19T12:01:59Z")
We introduced the `docker` prospector with Filebeat 6.1, specific for this use case: [https://www.elastic.co/guide/en/beats/filebeat/6.1/configuration-filebeat-options.html#config-containers](https://www.elastic.co/guide/en/beats/filebeat/6.1/configuration-filebeat-options.html#config-containers) The configuration to use it looks like: ```auto - type: docker containers.ids: - '*' processors: - add_docker_metadata: ~ ``` Then you can probably do the rest of processing from logstash, with a cleaner original message Best regards ---
**Author:** ![Asaf\_Shabat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asaf_shabat/32/24442_2.png) [@Asaf\_Shabat](https://discuss.elastic.co/u/Asaf_Shabat)\ **Post date:** [December 19, 2017, 1:10pm UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416/3 "2017-12-19T13:10:06Z")
Wow! you surprised me that you're supporting Docker logs OOTB now! But I'm returning to the main question, how can I parse the message log after it transferred from Filebeat to Logstash? I get the following message in Kibana: `e[0me[0m13:01:45.773 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) started in 23791ms - Started 794 of 1135 services (487 services are lazy, passive or on-demand)` What should I write in the logstash-filter.conf in order to parse the message? Currently I have this configuration in there: ``` filter { grok { match => { 'message' => '%{DATE:date} %{TIME:time} %{LOGLEVEL:loglevel}%{SPACE} \[(?[^\]]+)\] \((?[^\]]+)\)%{SPACE} %{GREEDYDATA:message}' } } } ``` and I'm unable to break the message above into fields like LOGLEVEL, LOGGER and etc... ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [December 19, 2017, 8:29pm UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416/4 "2017-12-19T20:29:12Z")
Hello, If I look at the message, I see color escape chars in the log `\u001b[0m\u001b[0m`, this is probably messing up your grok patterns, I would look at your configuration If it can probably be removed from the logs. After, I would either use the [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) to create the grok patterns OR I would switch to the [dissect filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html) which I believe should work well in your case and it's much easier to deal with and faster than grok. Thanks ---
**Author:** ![Asaf\_Shabat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asaf_shabat/32/24442_2.png) [@Asaf\_Shabat](https://discuss.elastic.co/u/Asaf_Shabat)\ **Post date:** [December 20, 2017, 7:36am UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416/5 "2017-12-20T07:36:36Z")
Thank you very much!! I changed the filter plugin to Dissect instead of Grok and it works well! This is the new filter pattern: > filter { > dissect { > mapping =\> { > "message" =\> "%{time} %{loglevel} [%{logger}] (%{thread}) %{message}" > } > } > } ---
**Author:** ![Asaf\_Shabat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asaf_shabat/32/24442_2.png) [@Asaf\_Shabat](https://discuss.elastic.co/u/Asaf_Shabat)\ **Post date:** [December 20, 2017, 2:19pm UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416/6 "2017-12-20T14:19:15Z")
Exekias, Does "docker type" module has the ability to compress the data? Currently, I'm using the "compression\_level: 9" in the Logstash section inside the filebeat.yml configuration file. Is the compression\_level applies to the docker type module? ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [January 2, 2018, 1:49pm UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416/7 "2018-01-02T13:49:33Z")
yes, `compression_level` parameter affects to all output, including docker prospector ---
**Author:** ![Asaf\_Shabat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asaf_shabat/32/24442_2.png) [@Asaf\_Shabat](https://discuss.elastic.co/u/Asaf_Shabat)\ **Post date:** [January 4, 2018, 12:45pm UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416/8 "2018-01-04T12:45:39Z")
Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 1, 2018, 12:45pm UTC](https://discuss.elastic.co/t/unable-to-parse-docker-json-file/112416/9 "2018-02-01T12:45:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.