Unable to drop fields in filebeat using drop_field

[vyjayanth]

Looking to drop a field called: Event.Original using drop_field. As Message Field produces same information as Event.Original.

I worked with remove_field of logstash filter, but it isn’t reflecting by dropping the field. Although it is nested representation of field, I was able to mention it as [event][original]. But it is not dropping the field.

Filebeat setting:

processors:
  - decode_json_fields:
      fields: ['event.original']
      process_array: false
      max_depth: 1
      target: ''
      overwrite_keys: false
      add_error_key: true
  - drop_fields:
      fields: ['event.original']

logstash Filter setting: Inside Grok

remove_field => ["[event][original]"]

However, this setting is not dropping Field as required. Does anyone have suggestions around it?

[Rios]

Welcome to the community!

Just put: fields: ['event']

[vyjayanth]

Hey, thanks.. I was looking to drop ‘original’field nested in side “event”… does it reflect…

[stephenb (Stephen Brown)]

Hi @vyjayanth

Welcome to the community

So you architecture is

Filebeat > Logstash> Elasticsearch?

What version?

Typically event.original is not created from filebeat It gets created by logstash or when the message arrives and goes through elasticsearch

That Would explain why you can't drop it at filebeat?

So tell us your architecture and the exact versions and share your logstash.

And then read about logstash ECS compatibility mode.

[vyjayanth]

Hi Stephen,

Yes, sorry that I didn’t mention before in the post.

We have a scenarios of Filebeat > Logstash > Splunk.

Filebeat : 8.8.0
Logstash: 8.8.2

Running as a side car container in rancher - k8.pod.

Oh ok, I was able to drop message field when I use drop-field feature in filebeat. Then I thought I could drop event.original.

I’ll bring in logstash conf for pipeline in a bit, not working today but outside…

[stephenb (Stephen Brown)]

@vyjayanth

Oh ok, I was able to drop message field when I use drop-field feature in filebeat.

You can only drop fields that are available at the time the are shipped from filebeat... event.original is most likely being created in logstash... not filebeat.

message field is the main field from filebeat, that is why you could drop it

[vyjayanth]

I can try it out…

[Rios]

If you need to remove in LS:
mutate{ remove_field => [ "event" ] }

Also if you disable ECS on the pipeline(pipelines.yml or for all ,confs in logstash.yml), you will not see that field.
pipeline.ecs_compatibility: disabled

[vyjayanth]

Hi Stephen and @Rios

I'm using the logstash setup for Springboot apps, here is the logstash's Filter Block. Please suggest any update, I'm new to GROK..

input {
    beats {
        port => 5044
    }
}

filter {
  if [fields][log-source] == "logfile" {
      grok {
          match => { "message" => "%{TIMESTAMP_ISO8601:time}:%{WORD:service} %{LOGLEVEL:loglevel} %{DATA:thread_name} \[%{DATA:src_file}:%{NUMBER:line}\] - %{GREEDYDATA:msg}" }
           }
       mutate {
          add_field => { "tmp_path" => "%{[log][file][path]}" }
        }
       mutate {
          split => ["tmp_path", "#"]
          add_field => { "kubernetes.node.name" => "%{tmp_path[2]}" }
          add_field => { "kubernetes.namespace" => "%{tmp_path[3]}" }
          add_field => { "tmp_podname" => "%{tmp_path[4]}" }
        }
       mutate {
          split => ["tmp_podname", "."]
          add_field => { "kubernetes.pod.name" => "%{tmp_podname[0]}" }
        }
       mutate {
          remove_field => [ "[event][original]", "tmp_path", "tmp_podname"]
        }
                                      }
        }

output {
            stdout {
                  codec => rubydebug
            }
            http {
                    format => "json"
                    content_type => "application/json"
                    http_method => "post"
                    url => "https://splunk-forwarder"
                    headers => ['Authorization', 'HEC']
                    ssl_verification_mode => none
            }
}

I initially tried to mention in Mutate { }, but with mutate{ remove_field => [ "[event][original]" ] } it made no difference. I just tried out mentioning "event" in mutate filter.

       mutate {
          remove_field => [ "event", "tmp_path", "tmp_podname"]
        }

It didn't drop the field.

[Rios]

Can you put a message sample?

[vyjayanth]

Hi @stephenb @Rios , it looks like disabling ecs has worked pretty well., it has dropped event{}.

Here is the event, after disabling ecs::

message: 2024-02-11 11:15:00,008 Application logger level info from .log file
input:
  type: log
ENV: DEV
ecs:
  version: 8.0.0
log:
  offset: 4331
  file:
    path: "/app-logs/schapp.log"
tags:
- beats_input_codec_plain_applied
agent:
  version: 8.8.0
  id: f8c6981e-6221-49fb-8b83-64c5a5e5a0e4
  name: podname-5d67c6789d-wz2v6
  type: filebeat
  ephemeral_id:  1234.1234.2314.5435.34534
"@version": '1'
NAMESPACE: <k8.namespacename>
host:
  name: podname-5d67c6789d-wz2v6
  os:
    platform: ubuntu
    name: Ubuntu
    kernel: 3.10.0-1160.105.1.el7.x86_64
    version: 20.04.6 LTS (Focal Fossa)
    type: linux
    codename: focal
    family: debian
  containerized: true
  hostname: <hostname>
  mac:
  - aa-ff-gg-hgh-hh
  ip:
  - xx.xx.xx.xx
  architecture: x86_64
"@timestamp": '2024-02-11T11:15:00.648Z'

p.s: changed the values fields

[Rios]

Just for test, move mutatete remove_field out of IF. It can for sure remove any field.

filter {
  if [fields][log-source] == "logfile"
  bla bla...
 } # end if
   mutate {
          remove_field => [ "event", "tmp_path", "tmp_podname"]
        }
} # filter end

Or use the prune filter:

      prune {
        blacklist_names => [ "^event$", "^tmp_" ]
      }

The parameter "pipeline.ecs_compatibility: disabled" doesn't use ECS however can be used as a workaround.