Unable to drop field from aws module

I have the following config:

- module: aws
    enabled: true
    var.queue_url: https://sqs.us-east-1.amazonaws.com/xxxxxxx/yyyyyyyyy
    enabled: false
    enabled: false
    enabled: false
    enabled: false
    enabled: false
- drop_fields:
    fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "event.original"]
    ignore_missing: true

All fields drop successfully, except event.original.

Hi @Ronin,

Filebeat events processing happens in two places when using modules: first, local processors are executed, then the event is sent to Elasticsearch, where another pipeline is executed. For the case of cloudtrail the pipeline in Elasticsearch is the one creating the event.original field (here). There is no option at the moment to remove this field.

There are two things you could try:

  • Use the S3 input directly instead of the cloudtrail module. This way you would have total control on the processing of events, but you would also be missing everything included in this module.
  • Modify the cloudtrail pipeline to remove the event.original field. Pipelines are installed by filebeat, and they are included in filebeat distributions, under /usr/share/filebeat/module/. You could modify it, and reinstall the pipeline with filebeat setup --pipelines --module aws. This has the problem that you will have to repeat the process every time you upgrade filebeat.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.