Unable to drop field from aws module

[Ronin]

I have the following config:

filebeat.modules:
- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: https://sqs.us-east-1.amazonaws.com/xxxxxxx/yyyyyyyyy
  cloudwatch:
    enabled: false
  ec2:
    enabled: false
  elb:
    enabled: false
  s3access:
    enabled: false
  vpcflow:
    enabled: false
processors:
- drop_fields:
    fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "event.original"]
    ignore_missing: true

All fields drop successfully, except event.original.

[jsoriano (Jaime Soriano)]

Hi @Ronin,

Filebeat events processing happens in two places when using modules: first, local processors are executed, then the event is sent to Elasticsearch, where another pipeline is executed. For the case of cloudtrail the pipeline in Elasticsearch is the one creating the event.original field (here). There is no option at the moment to remove this field.

There are two things you could try:

• Use the S3 input directly instead of the cloudtrail module. This way you would have total control on the processing of events, but you would also be missing everything included in this module.
• Modify the cloudtrail pipeline to remove the event.original field. Pipelines are installed by filebeat, and they are included in filebeat distributions, under /usr/share/filebeat/module/. You could modify it, and reinstall the pipeline with filebeat setup --pipelines --module aws. This has the problem that you will have to repeat the process every time you upgrade filebeat.