# Drop\_fields not working on doc **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 12, 2019, 1:05am UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545 "2019-12-12T01:05:52Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![brain](https://avatars.discourse-cdn.com/v4/letter/b/439d5e/32.png) [@brain](https://discuss.elastic.co/u/brain)\ **Post date:** [December 12, 2019, 1:05am UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545/1 "2019-12-12T01:05:52Z")
i'm trying to use drop\_fields on part of a document: ``` "signatures": [ { "markcount": 2, "families": [], "description": "The binary likely contains encrypted or compressed data indicative of a packer", "severity": 2, "marks": [ { "entropy": 7.90446336568116, "section": { "size_of_data": "0x00083400", "virtual_address": "0x0036d000", "entropy": 7.90446336568116, "name": "UPX1", "virtual_size": "0x00084000" }, "type": "generic", "description": "A section with a high entropy has been found" }, { "entropy": 0.9980988593155894, "type": "generic", "description": "Overall entropy of this PE file is high" } ],' ``` specifically, on signatures.marks.section, however my drop\_fields configuration does not work: ``` - drop_fields: fields: ["signatures.marks"] ``` i have tried signatures.marks.section, signatures.marks, signatures.marks etc to no avail. any ideas? ---
**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\ **Post date:** [December 12, 2019, 10:51am UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545/2 "2019-12-12T10:51:04Z")
hey @brain can you check the structure of whole reported event? is the message you mentioned already json decoded or is it as a string in a `source.message` field of the event. ---
**Author:** ![brain](https://avatars.discourse-cdn.com/v4/letter/b/439d5e/32.png) [@brain](https://discuss.elastic.co/u/brain)\ **Post date:** [December 12, 2019, 11:06am UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545/3 "2019-12-12T11:06:32Z")
the message is already json decoded. dropping the signatures field entirely works, but i'd prefer not to do that ---
**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\ **Post date:** [December 12, 2019, 11:23am UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545/4 "2019-12-12T11:23:45Z")
the problem is that you have an array of `signatures` you have few options here. first and ugly one is to provide enough remove entries so it will look like this ```auto - drop_event fields: ["signatures.[0].marks","signatures.[1].marks","signatures.[2].marks", etc.] ``` or use log stash (if you have one) and use ruby filter as discussed here: [Logstash filter mutate remove\_field inside of an array](https://discuss.elastic.co/t/logstash-filter-mutate-remove-field-inside-of-an-array/168503/11) ---
**Author:** ![brain](https://avatars.discourse-cdn.com/v4/letter/b/439d5e/32.png) [@brain](https://discuss.elastic.co/u/brain)\ **Post date:** [December 12, 2019, 11:28am UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545/5 "2019-12-12T11:28:26Z")
no logstash in the pipeline, so it'll be the ugly solution. how would that work with signatures.marks.section, which is also in an array? signatures.[0].marks.[0].section, signatures.[0].marks.[1].section, etc..? ---
**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\ **Post date:** [December 12, 2019, 11:31am UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545/6 "2019-12-12T11:31:28Z")
unfortunately yes. it depends on how many signatures you have in an array, and how many marks are inside a signature you need to go through [0,0],[0,1]...[0, marks], [1,0]... and so on unfortunately there's no nice way of doing it in a beat. for more advanced processing using logstash is advised ---
**Author:** ![brain](https://avatars.discourse-cdn.com/v4/letter/b/439d5e/32.png) [@brain](https://discuss.elastic.co/u/brain)\ **Post date:** [December 12, 2019, 11:45am UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545/8 "2019-12-12T11:45:32Z")
actually, spoke too soon. as a test i've configured: ``` - drop_fields: fields: ["signatures.[0].marks","signatures.[1].marks","signatures.[2].marks","signatures.[3].marks","signatures.[4].marks","signatures.[5].marks","signatures.[6].marks"] ignore_missing: true ``` however i'm still seeing the marks object in the doc, and the doc in question only has one signature field: # cat report.json | jq .signatures ``` [ { "markcount": 1, "families": [], "description": "The executable uses a known packer", "severity": 1, "marks": [ { "category": "packer", "ioc": "Armadillo v1.xx - v2.xx", "type": "ioc", "description": null } ], "references": [], "name": "peid_packer" } ] ``` ---
**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\ **Post date:** [December 16, 2019, 12:05pm UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545/9 "2019-12-16T12:05:14Z")
hey @brain I played with this a bit and I found out that beats does not support index pointers (something.0.somethinelse) this is supported only in LogStash you can specify path up to the point where you hit an array. having array as the first thing in your object is a bit unfortunate. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 13, 2020, 12:05pm UTC](https://discuss.elastic.co/t/drop-fields-not-working-on-doc/211545/10 "2020-01-13T12:05:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.