Mapper_parsing_exception, hashicorp vault audit, "error" in inner json

Hi!

Following Setup:
-> Docker 18.09
-> Filebeat 6.4.1 running in container with docker sock mounted
-> Hashicorp Vault running in container with audit enabled
-> extract from filebeat.yml

...
filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains.docker.container.name: "nginx-web"
          config:
            - module: nginx
              access:
                input:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"
        - condition:
            not.contains.docker.container.name: "nginx-web"
          config:
           - type: docker
             containers.ids:
                    - "${data.docker.container.id}"

processors:
- decode_json_fields:
      fields: ["message"]
      target: ""
      overwrite_keys: true
...

vault container writes following line on stdout (hashes are manipulated)

{
  "time": "2019-01-21T18:53:38.322674515Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:8d8185a5ec2e78206e4fasdfa8b648768c408a7faf85e52c5cf0248f",
    "accessor": "hmac-sha256:a8c7d8a1a951adfa7c65ef71870ef1f5a494d4330d41cbbf6a",
    "display_name": "root",
    "policies": [
      "root"
    ],
    "token_policies": [
      "root"
    ],
    "metadata": null,
    "entity_id": ""
  },
  "request": {
    "id": "3cdaf04d-b1c2-47a2-a7ca-9309379795bc",
    "operation": "update",
    "client_token": "hmac-sha256:8d8185a5ec2e7afd648768c408a7faf85e52c5cf0248f",
    "client_token_accessor": "hmac-sha256:a8c7d8a1aadfc65ef71870ef1f5a494d4330d41cbbf6a",
    "path": "sys/audit/file",
    "data": {
      "description": "hmac-sha256:1b079dfadf218abd5bab7f895ba073c55118c094e967d4",
      "local": false,
      "options": {
        "file_path": "hmac-sha256:b25edc3c72aadfd8d8909c7f65b99dd75bad5238fe3943dd53"
      },
      "type": "hmac-sha256:56bedf4adf1ec097f7b8da5c29d08db9749d86099d576aa1e"
    },
    "policy_override": false,
    "remote_address": "172.24.0.1",
    "wrap_ttl": 0,
    "headers": {}
  },
  "response": {},
  "error": ""
}

filebeat harvesting this log is not happy

2019-01-21T19:01:19.935Z WARN elasticsearch/client.go:520 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x2018b7b1, ext:63683694075, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"time":"2019-01-21T19:01:15.538248103Z", "type":"request", "request":map[string]interface {}{"headers":map[string]interface {}{}, "operation":"delete", "client_token":"hmac-sha256:8d8185a5ec2e78206e459a2edb22b15158b648768c408a7faf85e52c5cf0248f", "remote_address":"172.24.0.1", "wrap_ttl":0, "policy_override":false, "id":"8dd93fe4-6485-9e85-1cae-dcbfda4320e5", "client_token_accessor":"hmac-sha256:a8c7d8a1a9518271bea0e9e63b76277c65ef71870ef1f5a494d4330d41cbbf6a", "path":"sys/audit/file", "data":interface {}(nil)}, "source":"/var/lib/docker/containers/65f184d938d27867f7f64e6045b338f377722260a66344d800c8a700b60f042f/65f184d938d27867f7f64e6045b338f377722260a66344d800c8a700b60f042f-json.log", "offset":3209769, "input":common.MapStr{"type":"docker"}, "docker":common.MapStr{"container":common.MapStr{"name":"vault", "image":"vault:0.11.1", "labels":common.MapStr{}, "id":"65f184d938d27867f7f64e6045b338f377722260a66344d800c8a700b60f042f"}}, "host":common.MapStr{"name":"static-1-vm4"}, "stream":"stdout", "prospector":common.MapStr{"type":"docker"}, "message":"{\"time\":\"2019-01-21T19:01:15.538248103Z\",\"type\":\"request\",\"auth\":{\"client_token\":\"hmac-sha256:8d8185a5ec2e78206e459a2edb22b15158b648768c408a7faf85e52c5cf0248f\",\"accessor\":\"hmac-sha256:a8c7d8a1a9518271bea0e9e63b76277c65ef71870ef1f5a494d4330d41cbbf6a\",\"display_name\":\"root\",\"policies\":[\"root\"],\"token_policies\":[\"root\"],\"metadata\":null,\"entity_id\":\"\"},\"request\":{\"id\":\"8dd93fe4-6485-9e85-1cae-dcbfda4320e5\",\"operation\":\"delete\",\"client_token\":\"hmac-sha256:8d8185a5ec2e78206e459a2edb22b15158b648768c408a7faf85e52c5cf0248f\",\"client_token_accessor\":\"hmac-sha256:a8c7d8a1a9518271bea0e9e63b76277c65ef71870ef1f5a494d4330d41cbbf6a\",\"path\":\"sys/audit/file\",\"data\":null,\"policy_override\":false,\"remote_address\":\"172.24.0.1\",\"wrap_ttl\":0,\"headers\":{}},\"error\":\"\"}", "beat":common.MapStr{"name":"static-1-vm4", "hostname":"static-1-vm4", "version":"6.4.1"}, "auth":map[string]interface {}{"metadata":interface {}(nil), "entity_id":"", "client_token":"hmac-sha256:8d8185a5ec2e78206e459a2edb22b15158b648768c408a7faf85e52c5cf0248f", "accessor":"hmac-sha256:a8c7d8a1a9518271bea0e9e63b76277c65ef71870ef1f5a494d4330d41cbbf6a", "display_name":"root", "policies":[]interface {}{"root"}, "token_policies":[]interface {}{"root"}}, "error":""}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc420460dd0), Source:"/var/lib/docker/containers/65f184d938d27867f7f64e6045b338f377722260a66344d800c8a700b60f042f/65f184d938d27867f7f64e6045b338f377722260a66344d800c8a700b60f042f-json.log", Offset:3210665, Timestamp:time.Time{wall:0xbf0925b9583ae8b0, ext:6217344875309200, loc:(*time.Location)(0x1f61700)}, TTL:-1, Type:"docker", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x200be, Device:0x821}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [error] tried to parse field [error] as object, but found a concrete value"}

For clarification:

In the docker log json from the vault container is a message.
This message is also json (inner-json).
This json contains a field "error"
This field causes the parsing error.

type: mapper_parsing_exception
reason: object mapping for [error] tried to parse field [error] as object, but found a concrete value

I assume .. but really only assume, that this clashes with the "error" array in fields.yml coming from beats.

What can I do to make vault audit logging work with filebeat?

Due to overwrite_keys you have a clash between the filebeat internal event schema and the json document you are reading. This overwrites/sets the error field, which is incompatible to the schema used by beats.

First I'd not configure a global processor, as this will also try tp process the nginx modules events. Rather add the processors to the local not-nginx configs.

Setting the target field do a named value (maybe depending on container meta-data) will put the json into a separate namespace, reducing the chance of schema-clashes. You can also post-process your json by using the drop_fields or rename processors.

Full schema docs: https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields.html. The Beats fields define the error object. If you find 'error' to be a string, you could rename it to error.message.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.