# Removing fields with certain value (or if mapping is not found) from the output
**URL:** https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941
**Category:** Logstash
**Created:** [May 14, 2019, 7:43am UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941 "2019-05-14T07:43:30Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@saif3r](https://discuss.elastic.co/u/saif3r)
#### Post date: [May 14, 2019, 7:43am UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941/1 "2019-05-14T07:43:30Z")
Hello Guys,
I have a Logstash pipeline, responsible for pulling xml files. Each xml is split as suggested here by @Badger : [Turning one .xml into multiple events where not all values are always filled](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384)
Each field that has been split is mapped to a separate field within mutate, for example:
```
add_field => { "TEST_ID" => "%{[theXML][PRODUCT][0][GROUP][TEST][ID]}" }
add_field => { "TEST_NAME" => "%{[theXML][PRODUCT][0][GROUP][TEST][NAME]}" }
add_field => { "TEST_VALUE" => "%{[theXML][PRODUCT][0][GROUP][TEST][VALUE]}" }
add_field => { "TEST_STATUS" => "%{[theXML][PRODUCT][0][GROUP][TEST][STATUS]}" }
```
Unfortunately, each of those fields is optional, which means that they are not always there. In that case i end up with something like this in my index:
```
"TEST_ID": "1"
"TEST_NAME": "%{[theXML][PRODUCT][0][GROUP][TEST][NAME]}"
"TEST_VALUE": "102.66"
"TEST_STATUS" => "%{[theXML][PRODUCT][0][GROUP][TEST][STATUS]}"
```
Is it possible to set a rule saying that if a field starts with %{[theXML] or is not found in the source, it should be removed from the output? How could I handle this in the most efficient way?
Thanks in advance!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 14, 2019, 12:21pm UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941/2 "2019-05-14T12:21:49Z")
```
ruby {
code => '
event.to_hash.each { |k, v|
if v.to_s.start_with?("%{[theXML]")
event.remove(k)
end
}
'
}
```
---
### Author:  [@saif3r](https://discuss.elastic.co/u/saif3r)
#### Post date: [May 14, 2019, 12:26pm UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941/3 "2019-05-14T12:26:25Z")
Thank you @Badger. Seems to be working.
One more thing, is it possible to do the same with empty fields (those with value: "") within the same ruby script? I'm not familiar with ruby at all, hence the question.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 14, 2019, 12:29pm UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941/4 "2019-05-14T12:29:14Z")
Yes, you could just change the test to
```
if v == "" or v.to_s.start_with?("%{[theXML]")
```
---
### Author:  [@saif3r](https://discuss.elastic.co/u/saif3r)
#### Post date: [May 14, 2019, 12:41pm UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941/5 "2019-05-14T12:41:36Z")
Perfect, thank you.
---
### Author:  [@saif3r](https://discuss.elastic.co/u/saif3r)
#### Post date: [May 15, 2019, 10:50am UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941/6 "2019-05-15T10:50:20Z")
@Badger, slightly less related question. Is it possible to run ruby script to remove files based on a given string/regex? I'm asking because I would like to remove all the fields left over from initial xml split, so all [theXML] fields, including nested ones. There are plenty of those.
I can filter them out using Kibana's Source Filters, but i think it would be more effective to not send them at all. I tried to use prune but it removes all the fields before mapping so i cannot use them.
Thanks in advance.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 15, 2019, 12:48pm UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941/7 "2019-05-15T12:48:21Z")
If you want to remove [theXML] and all its sub-fields then just
```
mutate { remove_field => ["theXML"] }
```
Alternatively, when you are creating fields that you know you are going to remove later, just make them subfields of [@metadata]
---
### Author:  [@saif3r](https://discuss.elastic.co/u/saif3r)
#### Post date: [May 15, 2019, 1:21pm UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941/8 "2019-05-15T13:21:25Z")
@Badger
I cannot thank you enough. I owe you more than one bear at this point 🙂
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 12, 2019, 1:31pm UTC](https://discuss.elastic.co/t/removing-fields-with-certain-value-or-if-mapping-is-not-found-from-the-output/180941/9 "2019-06-12T13:31:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.