How to get rid of event.original field?

Elastic Stack Beats
1

After upgrading to ELK 8.1, I noticed that every event has the "event.original" field containing all of the log data. This is highly unwanted, how to prevent this field from being sent from Filebeat?
I tried doing it on Filebeat level using processors:

  processors:
  - drop_fields:
      fields: ["event.original"]

and via Logstash remove_field.
None of this worked, the field is still visible.

2

Are you using any filebeat module?

The field event.original is normally created by the ingest pipeline used by some of the filebeat modules when parsing the original message, it does not exist in your original event, so you won't be able to remove it in filebeat nor in logstash, you would need to check the ingest pipeline for the module that you are using and remove the field there.

3

My whole filebeat config is as below:

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
   - /path/to/IOB.log
  parsers:
  - multiline:
      type: pattern
      pattern: '^\[[0-9]{2}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]{2}:[0-9]{2}:[0-9]{2}'
      negate: True
      match: after
  name: iob1_int
  tags: ["iobint1-logs"]
  ignore_older: 24h
- type: filestream
  enabled: true
  paths:
   - /path/to/server.log
  parsers:
  - multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]{2}:[0-9]{2}:[0-9]{2}'
      negate: True
      match: after
  name: iob1_int
  tags: ["iobint1-logs_server"]
  ignore_older: 24h

processors:
  - drop_fields:
      fields: ["event.original","agent.type"]

output.logstash:
  hosts: ["hostname:5044"]
  ssl.certificate_authorities:
    - /path/to/ca.crt
monitoring:
  enabled: true
  cluster_uuid: 123456789
  elasticsearch:
    hosts: ["https://hostname:9200"]
    ssl.certificate_authorities: ["path/to/crt"]
    username: XXX
    password: XXX

After filebeat, Logstash parses the logs and sends to Elastic. Do you mean that the field is added on Logstash level? How to delete it?

4

What is your logstash pipeline? Please share it.

Logstash normally won't add any field unless explicitly configured in the pipeline, but I'm not running version 8.X and there was some changes regarding the ecs fields, so I'm not sure if this is being added by logstash or not.

How you tried to remove it in Logstash?

Did you have something like this?

mutate {
    remove_field => ["[event][original]"]
}
2 Likes
5

Thank you!
I tried to delete the field like below:
remove_field => ["event.original"]
and this didn't work.
This solves the issue for me.

6

Ah yes, this is confusing some times.

In filebeat and Elasticsearch you work with nested fields using top.nested, like event.original, but in logstash you need to use [top][nested], so it should be [event][original].

Using event.original in Logstash would make it try to work with a field with that literal name, with the dot in the name.

2 Likes
7

Thank you for a very fast response on a Friday afternoon! Much appreciated!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.