Filebeat issue with json logs from ES audit

Elastic Stack Beats
1

Hello,
I have following issue.
I created on latest Elasticsearch audit logs (ES ver 7.16.3) with json format,
and tried to impement filebeat to sending logs to ES.

My audit logs looks like as below:

{"type":"audit", "timestamp":"2022-02-16T12:17:44,600+0000", "node.name":"elasticsearch-tst-0", "node.id":"eGV6voTXTZ6zFqIXHcLvvQ", "event.type":"rest", "event.action":"authentication_success", "authentication.type":"REALM", "user.name":"elastic", "user.realm":"reserved", "origin.type":"rest", "origin.address":"172.17.0.4:58944", "realm":"reserved", "url.path":"/", "request.method":"GET", "request.id":"kXqxbClnRTisk8BaJRn5dg"}

The problem is that I receiving warnings from filebeat logs:
2022-02-16T12:19:49.428Z WARN [elasticsearch] elasticsearch/client.go:414 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.February, 16, 12, 19, 45, 999842859, time.Local), Meta:null, Fields:{"agent":{"ephemeral_id":"ed0edd5d-3b9e-461b-8ee4-baf511ca2ff8","hostname":"filebeat-tst-0","id":"a8d0d4b0-eddd-4aaa-8963-8bec71e7ccad","name":"filebeat-tst-0","type":"filebeat","version":"7.17.0"},"ecs":{"version":"1.12.0"},"event.action":"connection_granted","event.type":"ip_filter","host":{"architecture":"x86_64","containerized":true,"hostname":"filebeat-tst-0","ip":["172.17.0.5"],"mac":["02:42:ac:11:00:05"],"name":"filebeat-tst-0","os":{"codename":"focal","family":"debian","kernel":"5.13.0-28-lowlatency","name":"Ubuntu","platform":"ubuntu","type":"linux","version":"20.04.3 LTS (Focal Fossa)"}},"input":{"type":"log"},"log":{"file":{"path":"/opt/pci-cluster_audit.json"},"offset":743},"node.id":"eGV6voTXTZ6zFqIXHcLvvQ","node.name":"elasticsearch-tst-0","origin.address":"172.17.0.4","origin.type":"rest","rule":"allow default:accept_all","timestamp":"2022-02-16T12:17:44,638+0000","transport.profile":".http","type":"audit"}, Private:file.State{Id:"native::2072718-28", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000453ba0), Source:"/opt/pci-cluster_audit.json", Offset:1055, Timestamp:time.Date(2022, time.February, 16, 12, 19, 43, 18656250, time.Local), TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x1fa08e, Device:0x1c}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [rule] tried to parse field [rule] as object, but found a concrete value"}, dropping event!

My config of filebeat is as below:

    filebeat.inputs:
      - type: log
        paths:
          - /opt/*audit.json*
        json.keys_under_root: true
        json.add_error_key: true
        json.message_key: event.action
        ignore_older: 48h
        enabled: true
        scan_frequency: 100s
        close_inactive: 5m
        close_removed: true
    setup.ilm.enabled: false
    setup.template.name: audit-ecs-pci
    setup.template.pattern: audit-ecs-pci-*
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['http://172.17.0.2:9200']
      username: elastic
      password: "xxx"
      index: "audit-logs-%{+yyyy.MM.dd}"

Can You please tell me what is the issue that filebeat drops event ?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.