Filebeat 6.4.3 suddenly starts giving Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

Elastic Stack Beats
1

After rebooting filebeat docker on one of our servers it started giving this error for every log line:

Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

I can't really find what is causing this as all other servers seem to be running fine still. However, I can reproduce the issue locally in a test setup:

Filebeat Dockerfile:

FROM docker.elastic.co/beats/filebeat:6.4.3
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml

filebeat.yml:

filebeat.inputs:
  - type: docker
    #containers.ids: '*'
    containers.ids: ['e5601a3a6a11ee8857cd0edc600515e794e56b28d306fd139618c3060999169d']
    containers.stream: 'all'
    combine_partial: true
    multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    multiline.negate: true
    multiline.match: after
    ignore_older: 24h
    harvester_limit: 0
    json.keys_under_root: true
    json.add_error_key: true
    json.message_key: log

#processors:
#  - add_docker_metadata:
#      host: "unix:///var/run/docker.sock"

#output.logstash:
#  hosts: ["xxx:5044"]

output.console:
  enabled: true
  pretty: true

#xpack.monitoring:
#  enabled: true
#  elasticsearch:
#    hosts: ["xxx:9200"]

Log file located at /tmp/filebeat-docker-test/var/lib/docker/containers/e5601a3a6a11ee8857cd0edc600515e794e56b28d306fd139618c3060999169d/e5601a3a6a11ee8857cd0edc600515e794e56b28d306fd139618c3060999169d-json.log:

{"log":"2019-06-01T18:58:39.942Z\u0009ERROR\u0009json/json.go:51\u0009Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}\n","stream":"stderr","time":"2019-06-01T18:58:39.942960262Z"}
{"log":"2019-06-01T18:58:39.942Z\u0009ERROR\u0009json/json.go:51\u0009Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}\n","stream":"stderr","time":"2019-06-01T18:58:39.942960262Z"}

Build the docker container and run with: docker run --mount type=bind,source=/tmp/filebeat-docker-test/var/lib/docker,target=/var/lib/docker filebeat -e -d ""*

This post would get too long if I add the output, but the output showed the Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {} error once per log line. And the resulted documents do actually show the log field from the original log json.

I checked there is one JSON object per line. And as mentioned before, filebeat does seem to be running fine on other servers currently. But I can't really see whatever is different there, and am too afraid to restart anything there now..

Am I overlooking anything here?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.