# Filebeat failing to parse docker json-file logs

**URL:** <https://discuss.elastic.co/t/filebeat-failing-to-parse-docker-json-file-logs/297287>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [February 15, 2022, 6:40pm UTC](https://discuss.elastic.co/t/filebeat-failing-to-parse-docker-json-file-logs/297287 "2022-02-15T18:40:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rectalogic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rectalogic/32/44827_2.png) [@rectalogic](https://discuss.elastic.co/u/rectalogic)\
**Post date:** [February 15, 2022, 6:40pm UTC](https://discuss.elastic.co/t/filebeat-failing-to-parse-docker-json-file-logs/297287/1 "2022-02-15T18:40:03Z")

</div>

I'm using filebeat 7.12.0 to ingest docker logs using the container input. Log entries in the docker json-file format look like:

```auto
{"log":"{\"@timestamp\":\"2022-02-15T17:45:26.742Z\",\"log.level\":\"info\",\"message\":\"Auditor starting\",\"ecs\":{\"version\":\"1.6.0\"},\"log\":{\"logger\":\"cureatr.config.standalone\",\"origin\":{\"file\":{\"line\":237,\"name\":\"standalone.py\"},\"function\":\"process\"}},\"process\":{\"name\":\"MainProcess\",\"pid\":9,\"thread\":{\"id\":139770174854976,\"name\":\"MainThread\"}},\"service.name\":\"auditor\"}\n","stream":"stderr","time":"2022-02-15T17:45:26.743062038Z"}

```

Using this filebeat config:

```auto
- json.add_error_key: true
  json.expand_keys: true
  json.keys_under_root: true
  json.message_key: log
  json.overwrite_keys: true
  paths:
  - /site/docker/containers/*/*.log
  type: container

```

filebeat fails to parse the logs with this error:

```auto
2022-02-15T17:50:37.807Z ERROR [jsonhelper] jsontransform/jsonhelper.go:34 JSON: failed to expand fields: cannot expand "log.level": found conflicting key

```

If I comment out `json.expand_keys: true` then filebeat fails to parse the logs with:

```auto
2022-02-15T17:45:33.657Z WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x2c3a0580, ext:63780543926, loc:(*time.Location)(nil)}, Meta:null, Fields:{"agent":{"ephemeral_id":"f4f93b64-0f33-4d41-8cb8-a8130c20f7e6","hostname":"playtools1","id":"b6040b83-f25b-4729-9292-59cb4b7c73da","name":"playtools1","type":"filebeat","version":"7.12.0"},"ecs":{"version":"1.6.0"},"error":{"message":"Value of key 'log' is not a string","type":"json"},"host":{"name":"playtools1"},"input":{"type":"container"},"log":"","log.level":"info","message":"Auditor starting","process":{"name":"MainProcess","pid":9,"thread":{"id":139770174854976,"name":"MainThread"}},"service.name":"auditor","stream":"stderr"}, Private:file.State{Id:"native::534643-66306", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000117a00), Source:"/site/docker/containers/800b50f4f25eda132bc56cfc62746cfa2dff8af47852e1508f84fd0785f12c7b/800b50f4f25eda132bc56cfc62746cfa2dff8af47852e1508f84fd0785f12c7b-json.log", Offset:986, Timestamp:time.Time{wall:0xc07b178f3b188019, ext:310110818695, loc:(*time.Location)(0x4163ac0)}, TTL:-1, Type:"container", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x82873, Device:0x10302}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [log] tried to parse field [log] as object, but found a concrete value"}

```

Is there any way to make filebeat parse these logs as json and expand keys?

---

<div class="post-metadata">

**Author:** ![rectalogic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rectalogic/32/44827_2.png) [@rectalogic](https://discuss.elastic.co/u/rectalogic)\
**Post date:** [February 15, 2022, 8:48pm UTC](https://discuss.elastic.co/t/filebeat-failing-to-parse-docker-json-file-logs/297287/2 "2022-02-15T20:48:35Z")

</div>

OK, removing `json.message_key: log` fixed it - I guess the `container` input already knows the message is under the `log` key.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 15, 2022, 10:48pm UTC](https://discuss.elastic.co/t/filebeat-failing-to-parse-docker-json-file-logs/297287/3 "2022-03-15T22:48:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.