# Problem with "docker" prospector and "error decoding json" logs

**URL:** <https://discuss.elastic.co/t/problem-with-docker-prospector-and-error-decoding-json-logs/132828>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 22, 2018, 1:11pm UTC](https://discuss.elastic.co/t/problem-with-docker-prospector-and-error-decoding-json-logs/132828 "2018-05-22T13:11:45Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![LollerAgent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lolleragent/32/47486_2.png) [@LollerAgent](https://discuss.elastic.co/u/LollerAgent)\
**Post date:** [May 22, 2018, 1:11pm UTC](https://discuss.elastic.co/t/problem-with-docker-prospector-and-error-decoding-json-logs/132828/1 "2018-05-22T13:11:45Z")

</div>

When using the `docker` prospector and specifying `containers.ids: *` my logs are constantly spammed with:

```auto
2018-05-21T17:39:31.019Z	ERROR	reader/json.go:32	Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

```

I'm a bit confused because all of my logs are generated by Docker's `json-logs` driver which outputs valid JSON. The logs are killing filebeat. I understand the `json.ignore_decoding_error` option was introduced in 6.3 - but we are currently using 6.2.4 and I don't see published Docker images for 6.3 yet.

If I use the `log` prospector instead of `docker` and specify the oath as `/var/lib/docker/container/*/*.log`, the errors go away.

Here is an example log generated by Docker:

```auto
{"legacy_message":"ts=2018-05-22T13:02:06.658714068Z method=GET path=/stylesheets/fonts/WhitneySSm-BookItalic.ttf host= remote_ip=173.197.145.174","message":"","level":"info","distributed_trace_id":"","http":{"request":{"method":"GET","path":"/stylesheets/fonts/WhitneySSm-BookItalic.ttf","host":"","remote_ip":"173.197.145.174"},"response":{"status_code":404}}}

```

Is there any workaround in 6.2.4?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 22, 2018, 3:25pm UTC](https://discuss.elastic.co/t/problem-with-docker-prospector-and-error-decoding-json-logs/132828/2 "2018-05-22T15:25:28Z")

</div>

Maybe the problem is not in the logs but in the configuration, `container.ids` value needs to be a list, so in your case you would need to specify `containers.ids: [*]`, try to change this.

---

<div class="post-metadata">

**Author:** ![jeffspahr](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@jeffspahr](https://discuss.elastic.co/u/jeffspahr)\
**Post date:** [June 12, 2018, 4:53pm UTC](https://discuss.elastic.co/t/problem-with-docker-prospector-and-error-decoding-json-logs/132828/3 "2018-06-12T16:53:06Z")

</div>

> [@jsoriano](#):
>
> Maybe the problem is not in the logs but in the configuration, `container.ids` value needs to be a list, so in your case you would need to specify `containers.ids: [*]` , try to change this.

I ran into the same issue. Changing to `containers.ids: [*]` causes the following error:  
2018-06-12T16:48:31.126Z ERROR cfgfile/reload.go:237 Error loading config: invalid config: yaml: line 1: did not find expected alphabetic or numeric character

Also all the documentation points to containers.ids: '_' or:  
containers.ids:  
- "_"  
being correct.  
[https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-docker.html#config-container-ids](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-docker.html#config-container-ids)  
[Run Filebeat on Kubernetes | Filebeat Reference [master] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/master/running-on-kubernetes.html) - Look at the config in the k8s manifest.

---

<div class="post-metadata">

**Author:** ![jeffspahr](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@jeffspahr](https://discuss.elastic.co/u/jeffspahr)\
**Post date:** [June 12, 2018, 4:54pm UTC](https://discuss.elastic.co/t/problem-with-docker-prospector-and-error-decoding-json-logs/132828/4 "2018-06-12T16:54:15Z")

</div>

I'll also add that my config worked on k8s 1.7 (OpenShift 3.7), and I believe it stopped working after our upgrade to k8s 1.9 (OpenShift 3.9).

---

<div class="post-metadata">

**Author:** ![LollerAgent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lolleragent/32/47486_2.png) [@LollerAgent](https://discuss.elastic.co/u/LollerAgent)\
**Post date:** [June 12, 2018, 5:32pm UTC](https://discuss.elastic.co/t/problem-with-docker-prospector-and-error-decoding-json-logs/132828/5 "2018-06-12T17:32:00Z")

</div>

We ended up switching back to the "log" prospector:

```auto
filebeat.prospectors:
  # Must use "log" prospector (instead of "docker") until 6.3 is released due to JSON errors
  # The "docker" prospector also sets defaults for the json. values so they will need to be modified
  # https://github.com/elastic/beats/issues/6045
- type: log
  paths:
   - '/var/lib/docker/containers/*/*.log'
  # Hard-setting this to false causes decoded JSON fields to be under the "json" key
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: log
  processors:
  - add_docker_metadata: ~
  - decode_json_fields:
      fields: ["log"]

```

If you are having problems, I would try using a similar configuration.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 10, 2018, 5:32pm UTC](https://discuss.elastic.co/t/problem-with-docker-prospector-and-error-decoding-json-logs/132828/6 "2018-07-10T17:32:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.