Problem with "docker" prospector and "error decoding json" logs

Elastic Stack Beats
1

When using the docker prospector and specifying containers.ids: * my logs are constantly spammed with:

2018-05-21T17:39:31.019Z	ERROR	reader/json.go:32	Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

I'm a bit confused because all of my logs are generated by Docker's json-logs driver which outputs valid JSON. The logs are killing filebeat. I understand the json.ignore_decoding_error option was introduced in 6.3 - but we are currently using 6.2.4 and I don't see published Docker images for 6.3 yet.

If I use the log prospector instead of docker and specify the oath as /var/lib/docker/container/*/*.log, the errors go away.

Here is an example log generated by Docker:

{"legacy_message":"ts=2018-05-22T13:02:06.658714068Z method=GET path=/stylesheets/fonts/WhitneySSm-BookItalic.ttf host= remote_ip=173.197.145.174","message":"","level":"info","distributed_trace_id":"","http":{"request":{"method":"GET","path":"/stylesheets/fonts/WhitneySSm-BookItalic.ttf","host":"","remote_ip":"173.197.145.174"},"response":{"status_code":404}}}

Is there any workaround in 6.2.4?

2

Maybe the problem is not in the logs but in the configuration, container.ids value needs to be a list, so in your case you would need to specify containers.ids: [*], try to change this.

3

I ran into the same issue. Changing to containers.ids: [*] causes the following error:
2018-06-12T16:48:31.126Z ERROR cfgfile/reload.go:237 Error loading config: invalid config: yaml: line 1: did not find expected alphabetic or numeric character

Also all the documentation points to containers.ids: '' or:
containers.ids:
- ""
being correct.
https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-docker.html#config-container-ids
Run Filebeat on Kubernetes | Filebeat Reference [master] | Elastic - Look at the config in the k8s manifest.

4

I'll also add that my config worked on k8s 1.7 (OpenShift 3.7), and I believe it stopped working after our upgrade to k8s 1.9 (OpenShift 3.9).

5

We ended up switching back to the "log" prospector:

filebeat.prospectors:
  # Must use "log" prospector (instead of "docker") until 6.3 is released due to JSON errors
  # The "docker" prospector also sets defaults for the json. values so they will need to be modified
  # https://github.com/elastic/beats/issues/6045
- type: log
  paths:
   - '/var/lib/docker/containers/*/*.log'
  # Hard-setting this to false causes decoded JSON fields to be under the "json" key
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: log
  processors:
  - add_docker_metadata: ~
  - decode_json_fields:
      fields: ["log"]

If you are having problems, I would try using a similar configuration.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.