Filebeat 7.1.0 Elasticsearch module audit pipeline incorrectly references @timestamp instead of timestamp

Elastic Stack Beats
1

I'm using Filebeat and Elasticsearch 7.1.0 to capture logs through the Docker input and parsing with the elasticsearch module with config such as:

filebeat.autodiscover:
  providers:
    # elasticsearch
    - type: docker
      labels.dedot: true
      templates:
        - condition:
            contains:
              docker.container.image: elasticsearch
          config:
            - module: elasticsearch
              audit:
                enabled: true
                input:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"
                  exclude_files: ['\.gz$']
                  containers.stream: stdout
                  include_lines: ['"type": "audit"']

This is successfully sending logs from my elasticsearch-based Docker services to my elasticsearch cluster, however I see the following error in all audit records:

     "error": {
      "message": "field [@timestamp] not present as part of path [elasticsearch.audit.@timestamp]"
    }

Looking at the ingest pipeline installed into Elasticsearch by Filebeat (_ingest/pipeline/filebeat-7.1.0-elasticsearch-audit-pipeline), I believe the problem is:

{
  "date": {
    "field": "elasticsearch.audit.@timestamp",
    "target_field": "@timestamp",
    "formats": [
      "ISO8601"
    ],
    "ignore_failure": true
  }
},
{
  "remove": {
    "field": "elasticsearch.audit.@timestamp"
  }
},

Should instead match the similar lines from the server pipeline (_ingest/pipeline/filebeat-7.1.0-elasticsearch-server-pipeline):

{
  "date": {
    "field": "elasticsearch.server.timestamp",
    "target_field": "@timestamp",
    "formats": [
      "ISO8601"
    ],
    "ignore_failure": true
  }
},
{
  "remove": {
    "field": "elasticsearch.server.timestamp"
  }
},

This appears to be a problem in the Filebeat elasticsearch audit pipeline definition.

I think this may also be why I don't see a lot of the logs that have been processed by the elasticsearch module when using the default Filebeat-installed index template and pattern (there is no top-level "@timestamp" field defined for a lot of the documents, which prevents them showing up in the "filebeat-*" queries because that field is specified as the time filter field).

N.B. deprecation and slowlog pipelines both appear to be correct (like server).

2

Hi @Chris_Samo,

Welcome to the Elastic community forums and thanks for posting such a detailed description of this issue!

Interestingly, when Elasticsearch audit logging is turned on in a non-Docker environment, the audit log entries look like this:

{"@timestamp":"2019-05-28T12:33:31,246", "node.id":"qUZG15diSquz8jpvQbTmhA", "event.type":"rest", "event.action":"anonymous_access_denied", "origin.type":"rest", "origin.address":"[::1]:49499", "url.path":"/", "request.method":"GET", "request.id":"b5A_F-ihQaOHrxBgAa3gjA"}

Note the name of the timestamp field here: it is @timestamp, not timestamp. That's why the Filebeat ingest pipeline is currently looking for @timestamp.

Of course, for deprecation, server, and slow logs, the timestamp field is named timestamp in Docker and non-Docker contexts so the Filebeat ingest pipelines work in those cases.

I'm looking into the naming inconsistency for the audit log's timestamp field and will post an update here.

Thanks,

Shaunak

3

@shaunak thanks for the quick reply, I'll keep an eye on future updates to the Dockerised version of Filebeat for any changes (and continue to use my workaround for the time being while we figure out the best way to index and visualise the data).

4

Small update: after discussing this with Elasticsearch developers, we've created two GitHub issues to track a short-term fix in Filebeat and a longer-term fix in Elasticsearch:

If you have an account on GitHub you may want to subscribe to these issues and receive updates on their progress.

Thanks,

Shaunak

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.