Getting "failed to parse field [response.body] of type [keyword] in document..."

[zhelezovas (Артём Железов)]

Hello, I'm trying to parse json logs with filebeat processor. In general it works fine, but sometimes I get "failed to parse field [response.body] of type [keyword] in document..." error.

This is my filebeat config file:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      host: ${NODE_NAME}
      templates:
        - config:
          - type: docker
            containers.ids:
              - "${data.kubernetes.container.id}"
            processors:
              - decode_json_fields:
                  fields: ["message"]
                  target: ""
                  overwrite_keys: true
                  max_depth: 8

setup.ilm.enabled: false
setup.template.name: "index-"
setup.template.pattern: "index-*"
output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
  index: "index-dev-%{[kubernetes.namespace]}-%{[agent.version]}-%{+yyyy.MM}"

Filebeat can't parse this log:

{"level":30,"time":1575295869865,"pid":1,"hostname":"crud-769d6cc499-m2w4k","request":{"from":"10.0.4.8","to":"GET /crud/1.0/data","headers":{"host":"sandbox.dev.ladcloud.ru","x-request-id":"a5bd4fef58baedb732cfd72e240b2f84","x-real-ip":"95.79.56.97","x-forwarded-for":"95.79.56.97","x-forwarded-host":"sandbox.dev.ladcloud.ru","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/crud/1.0/data?userId=d69bfbdd-ffbb-4225-8851-7155bcb3684f","x-scheme":"https","user-agent":"curl/7.58.0","accept":"*/*","authorization":"Bearer c4274f7c-8ac3-42ce-b836-f2a118805936"},"body":null},"response":{"body":{"statusCode":503,"error":"Service Unavailable","message":"Jopa lala"},"statusCode":503},"v":1}

Error message:

2019-12-02T14:11:11.395Z	WARN	elasticsearch/client.go:535	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x339cf53a, ext:63710892669, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"7d023a40-f4a6-4475-a2aa-76d858697445", "hostname":"k8s-dev-worker-1", "id":"c72a725a-f193-484f-bc2c-4da3f0f3c35f", "type":"filebeat", "version":"7.3.0"}, "ecs":common.MapStr{"version":"1.0.1"}, "host":common.MapStr{"name":"k8s-dev-worker-1"}, "hostname":"crud-769d6cc499-m2w4k", "input":common.MapStr{"type":"docker"}, "kubernetes":common.MapStr{"container":common.MapStr{"name":"crud"}, "labels":common.MapStr{"app":"crud", "pod-template-hash":"769d6cc499"}, "namespace":"sandbox", "node":common.MapStr{"name":"k8s-dev-worker-1"}, "pod":common.MapStr{"name":"crud-769d6cc499-m2w4k", "uid":"66210803-c6d7-40e5-af23-4b159c37fc07"}, "replicaset":common.MapStr{"name":"crud-769d6cc499"}}, "level":30, "log":common.MapStr{"file":common.MapStr{"path":"/var/lib/docker/containers/62d4ce6cc6531ec3485968b1f30ed351588e9d9ef4e20e9cd4b90eb24a84ee95/62d4ce6cc6531ec3485968b1f30ed351588e9d9ef4e20e9cd4b90eb24a84ee95-json.log"}, "offset":49720573}, "message":"{\"level\":30,\"time\":1575295869865,\"pid\":1,\"hostname\":\"crud-769d6cc499-m2w4k\",\"request\":{\"from\":\"10.0.4.8\",\"to\":\"GET /crud/1.0/data\",\"headers\":{\"host\":\"sandbox.dev.ladcloud.ru\",\"x-request-id\":\"a5bd4fef58baedb732cfd72e240b2f84\",\"x-real-ip\":\"95.79.56.97\",\"x-forwarded-for\":\"95.79.56.97\",\"x-forwarded-host\":\"sandbox.dev.ladcloud.ru\",\"x-forwarded-port\":\"443\",\"x-forwarded-proto\":\"https\",\"x-original-uri\":\"/crud/1.0/data?userId=d69bfbdd-ffbb-4225-8851-7155bcb3684f\",\"x-scheme\":\"https\",\"user-agent\":\"curl/7.58.0\",\"accept\":\"*/*\",\"authorization\":\"Bearer c4274f7c-8ac3-42ce-b836-f2a118805936\"},\"body\":null},\"response\":{\"body\":{\"statusCode\":503,\"error\":\"Service Unavailable\",\"message\":\"Jopa lala\"},\"statusCode\":503},\"v\":1}", "pid":1, "request":map[string]interface {}{"body":interface {}(nil), "from":"10.0.4.8", "headers":map[string]interface {}{"accept":"*/*", "authorization":"Bearer c4274f7c-8ac3-42ce-b836-f2a118805936", "host":"sandbox.dev.ladcloud.ru", "user-agent":"curl/7.58.0", "x-forwarded-for":"95.79.56.97", "x-forwarded-host":"sandbox.dev.ladcloud.ru", "x-forwarded-port":"443", "x-forwarded-proto":"https", "x-original-uri":"/crud/1.0/data?userId=d69bfbdd-ffbb-4225-8851-7155bcb3684f", "x-real-ip":"95.79.56.97", "x-request-id":"a5bd4fef58baedb732cfd72e240b2f84", "x-scheme":"https"}, "to":"GET /crud/1.0/data"}, "response":map[string]interface {}{"body":map[string]interface {}{"error":"Service Unavailable", "message":"Jopa lala", "statusCode":503}, "statusCode":503}, "stream":"stdout", "time":1575295869865, "v":1}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000375c70), Source:"/var/lib/docker/containers/62d4ce6cc6531ec3485968b1f30ed351588e9d9ef4e20e9cd4b90eb24a84ee95/62d4ce6cc6531ec3485968b1f30ed351588e9d9ef4e20e9cd4b90eb24a84ee95-json.log", Offset:49721443, Timestamp:time.Time{wall:0xbf7161519bff1c3b, ext:53337923, loc:(*time.Location)(0x30d3480)}, TTL:-1, Type:"docker", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xa12d60, Device:0x801}}, TimeSeries:false}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [response.body] of type [keyword] in document with id '8ybzxm4BUWUc5hmsaYmR'. Preview of field's value: '{error=Service Unavailable, message=Jopa lala, statusCode=503}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:61"}}

Any help appreciated.