Problem with "docker" prospector and "error decoding json" logs


When using the docker prospector and specifying containers.ids: * my logs are constantly spammed with:

2018-05-21T17:39:31.019Z	ERROR	reader/json.go:32	Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

I'm a bit confused because all of my logs are generated by Docker's json-logs driver which outputs valid JSON. The logs are killing filebeat. I understand the json.ignore_decoding_error option was introduced in 6.3 - but we are currently using 6.2.4 and I don't see published Docker images for 6.3 yet.

If I use the log prospector instead of docker and specify the oath as /var/lib/docker/container/*/*.log, the errors go away.

Here is an example log generated by Docker:

{"legacy_message":"ts=2018-05-22T13:02:06.658714068Z method=GET path=/stylesheets/fonts/WhitneySSm-BookItalic.ttf host= remote_ip=","message":"","level":"info","distributed_trace_id":"","http":{"request":{"method":"GET","path":"/stylesheets/fonts/WhitneySSm-BookItalic.ttf","host":"","remote_ip":""},"response":{"status_code":404}}}

Is there any workaround in 6.2.4?

(Jaime Soriano) #2

Maybe the problem is not in the logs but in the configuration, container.ids value needs to be a list, so in your case you would need to specify containers.ids: [*], try to change this.

(Jeffspahr) #3

I ran into the same issue. Changing to containers.ids: [*] causes the following error:
2018-06-12T16:48:31.126Z ERROR cfgfile/reload.go:237 Error loading config: invalid config: yaml: line 1: did not find expected alphabetic or numeric character

Also all the documentation points to containers.ids: '' or:
- "
being correct. - Look at the config in the k8s manifest.

(Jeffspahr) #4

I'll also add that my config worked on k8s 1.7 (OpenShift 3.7), and I believe it stopped working after our upgrade to k8s 1.9 (OpenShift 3.9).


We ended up switching back to the "log" prospector:

  # Must use "log" prospector (instead of "docker") until 6.3 is released due to JSON errors
  # The "docker" prospector also sets defaults for the json. values so they will need to be modified
- type: log
   - '/var/lib/docker/containers/*/*.log'
  # Hard-setting this to false causes decoded JSON fields to be under the "json" key
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: log
  - add_docker_metadata: ~
  - decode_json_fields:
      fields: ["log"]

If you are having problems, I would try using a similar configuration.

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.