# Unable to drop fields in filebeat using drop\_field **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 9, 2024, 9:57pm UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978 "2024-02-09T21:57:34Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![vyjayanth](https://avatars.discourse-cdn.com/v4/letter/v/b9e5f3/32.png) [@vyjayanth](https://discuss.elastic.co/u/vyjayanth)\ **Post date:** [February 9, 2024, 9:57pm UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/1 "2024-02-09T21:57:34Z")
Looking to drop a field called: Event.Original using drop\_field. As Message Field produces same information as Event.Original. I worked with remove\_field of logstash filter, but it isn’t reflecting by dropping the field. Although it is nested representation of field, I was able to mention it as [event][original]. But it is not dropping the field. Filebeat setting: ```auto processors: - decode_json_fields: fields: ['event.original'] process_array: false max_depth: 1 target: '' overwrite_keys: false add_error_key: true - drop_fields: fields: ['event.original'] ``` logstash Filter setting: Inside Grok ```auto remove_field => ["[event][original]"] ``` However, this setting is not dropping Field as required. Does anyone have suggestions around it? ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [February 10, 2024, 7:25am UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/2 "2024-02-10T07:25:13Z")
Welcome to the community! Just put: `fields: ['event']` ---
**Author:** ![vyjayanth](https://avatars.discourse-cdn.com/v4/letter/v/b9e5f3/32.png) [@vyjayanth](https://discuss.elastic.co/u/vyjayanth)\ **Post date:** [February 10, 2024, 9:36pm UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/3 "2024-02-10T21:36:58Z")
Hey, thanks.. I was looking to drop ‘original’field nested in side “event”… does it reflect… ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 10, 2024, 10:07pm UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/4 "2024-02-10T22:07:01Z")
Hi @vyjayanth Welcome to the community So you architecture is Filebeat \> Logstash\> Elasticsearch? What version? Typically `event.original` is not created from filebeat It gets created by logstash or when the message arrives and goes through elasticsearch That Would explain why you can't drop it at filebeat? So tell us your architecture and the exact versions and share your logstash. And then read about logstash ECS compatibility mode. ---
**Author:** ![vyjayanth](https://avatars.discourse-cdn.com/v4/letter/v/b9e5f3/32.png) [@vyjayanth](https://discuss.elastic.co/u/vyjayanth)\ **Post date:** [February 10, 2024, 10:35pm UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/5 "2024-02-10T22:35:14Z")
Hi Stephen, Yes, sorry that I didn’t mention before in the post. We have a scenarios of Filebeat \> Logstash \> Splunk. Filebeat : 8.8.0 Logstash: 8.8.2 Running as a side car container in rancher - k8.pod. Oh ok, I was able to drop message field when I use drop-field feature in filebeat. Then I thought I could drop event.original. I’ll bring in logstash conf for pipeline in a bit, not working today but outside… ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 11, 2024, 1:01am UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/6 "2024-02-11T01:01:52Z")
> [@vyjayanth](#): > > Oh ok, I was able to drop message field when I use drop-field feature in filebeat. You can only drop fields that are available at the time the are shipped from filebeat... `event.original` is most likely being created in logstash... not filebeat. `message` field is the main field from filebeat, that is why you could drop it ---
**Author:** ![vyjayanth](https://avatars.discourse-cdn.com/v4/letter/v/b9e5f3/32.png) [@vyjayanth](https://discuss.elastic.co/u/vyjayanth)\ **Post date:** [February 11, 2024, 1:47am UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/7 "2024-02-11T01:47:39Z")
I can try it out… ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [February 11, 2024, 4:04am UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/8 "2024-02-11T04:04:44Z")
If you need to remove in LS: ` mutate{ remove_field => ["event"] }` Also if you disable ECS on the pipeline(pipelines.yml or for all ,confs in logstash.yml), you will not see that field. `pipeline.ecs_compatibility: disabled` ---
**Author:** ![vyjayanth](https://avatars.discourse-cdn.com/v4/letter/v/b9e5f3/32.png) [@vyjayanth](https://discuss.elastic.co/u/vyjayanth)\ **Post date:** [February 11, 2024, 10:24am UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/9 "2024-02-11T10:24:06Z")
Hi Stephen and @Rios I'm using the logstash setup for Springboot apps, here is the logstash's Filter Block. Please suggest any update, I'm new to GROK.. ```auto input { beats { port => 5044 } } filter { if [fields][log-source] == "logfile" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:time}:%{WORD:service} %{LOGLEVEL:loglevel} %{DATA:thread_name} \[%{DATA:src_file}:%{NUMBER:line}\] - %{GREEDYDATA:msg}" } } mutate { add_field => { "tmp_path" => "%{[log][file][path]}" } } mutate { split => ["tmp_path", "#"] add_field => { "kubernetes.node.name" => "%{tmp_path[2]}" } add_field => { "kubernetes.namespace" => "%{tmp_path[3]}" } add_field => { "tmp_podname" => "%{tmp_path[4]}" } } mutate { split => ["tmp_podname", "."] add_field => { "kubernetes.pod.name" => "%{tmp_podname[0]}" } } mutate { remove_field => ["[event][original]", "tmp_path", "tmp_podname"] } } } output { stdout { codec => rubydebug } http { format => "json" content_type => "application/json" http_method => "post" url => "https://splunk-forwarder" headers => ['Authorization', 'HEC'] ssl_verification_mode => none } } ``` I initially tried to mention in `Mutate { }`, but with `mutate{ remove_field => ["[event][original]" ] }` it made no difference. I just tried out mentioning "event" in mutate filter. ```auto mutate { remove_field => ["event", "tmp_path", "tmp_podname"] } ``` It didn't drop the field. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [February 11, 2024, 11:08am UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/10 "2024-02-11T11:08:30Z")
Can you put a message sample? ---
**Author:** ![vyjayanth](https://avatars.discourse-cdn.com/v4/letter/v/b9e5f3/32.png) [@vyjayanth](https://discuss.elastic.co/u/vyjayanth)\ **Post date:** [February 11, 2024, 11:39am UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/11 "2024-02-11T11:39:39Z")
Hi @stephenb @Rios , it looks like disabling ecs has worked pretty well., it has dropped event{}. Here is the event, after disabling ecs:: ```auto message: 2024-02-11 11:15:00,008 Application logger level info from .log file input: type: log ENV: DEV ecs: version: 8.0.0 log: offset: 4331 file: path: "/app-logs/schapp.log" tags: - beats_input_codec_plain_applied agent: version: 8.8.0 id: f8c6981e-6221-49fb-8b83-64c5a5e5a0e4 name: podname-5d67c6789d-wz2v6 type: filebeat ephemeral_id: 1234.1234.2314.5435.34534 "@version": '1' NAMESPACE: host: name: podname-5d67c6789d-wz2v6 os: platform: ubuntu name: Ubuntu kernel: 3.10.0-1160.105.1.el7.x86_64 version: 20.04.6 LTS (Focal Fossa) type: linux codename: focal family: debian containerized: true hostname: mac: - aa-ff-gg-hgh-hh ip: - xx.xx.xx.xx architecture: x86_64 "@timestamp": '2024-02-11T11:15:00.648Z' ``` p.s: changed the values fields ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [February 11, 2024, 1:56pm UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/12 "2024-02-11T13:56:22Z")
Just for test, move mutatete remove\_field out of IF. It can for sure remove any field. ```auto filter { if [fields][log-source] == "logfile" bla bla... } # end if mutate { remove_field => ["event", "tmp_path", "tmp_podname"] } } # filter end ``` Or use the prune filter: ```auto prune { blacklist_names => ["^event$", "^tmp_"] } ``` The parameter "pipeline.ecs\_compatibility: disabled" doesn't use [ECS](https://www.elastic.co/guide/en/ecs/current/index.html) however can be used as a workaround. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 10, 2024, 3:56pm UTC](https://discuss.elastic.co/t/unable-to-drop-fields-in-filebeat-using-drop-field/352978/13 "2024-03-10T15:56:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.