# Delete inner fields (from json) using lookup file

**URL:** <https://discuss.elastic.co/t/delete-inner-fields-from-json-using-lookup-file/41294>\
**Category:** Logstash\
**Created:** [February 9, 2016, 3:45pm UTC](https://discuss.elastic.co/t/delete-inner-fields-from-json-using-lookup-file/41294 "2016-02-09T15:45:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kryten](https://avatars.discourse-cdn.com/v4/letter/k/58956e/32.png) [@Kryten](https://discuss.elastic.co/u/Kryten)\
**Post date:** [February 9, 2016, 3:45pm UTC](https://discuss.elastic.co/t/delete-inner-fields-from-json-using-lookup-file/41294/1 "2016-02-09T15:45:19Z")

</div>

Hi,

I'm processing quite large files using the json input codec. It's doing a great job of splitting up the data, creating fields and passing it to ES.

However... Some of the documents have a nested value a bit like this:

```
{ .... <stuff up here ... ,
    "outer" : { 
        "inner_data_one" : val1,
        "inner_data_two": val2,
        "inner_data_three": val3
}

```

This, quite correctly results in logstash creating fields like:

```
"outer" => {
    "inner_data_one" => val1,
    "inner_data_two" => val2,
    "inner_data_three" => val3
}

```

Which appear in ES as outer.inner\_data\_one and so on...

I need a way to delete SOME of those inner values... so far I see that I can do :

```
if [outer] {
		mutate {
			remove_field => ["[outer][inner_data_one]"]
               }

```

This works and does remove the inner\_data\_one field which is great.. BUT... it is becoming quite laborious keeping the config up to date with potentially scores of inner data fields to be removed.

What I am looking for is a way to look up all the inner field names in a .txt file and if they are found there for them to be removed.

Is this possible?

Many thanks.

PS -- Re-reading this post I suppose what I might really be asking, which is simpler, is can remove\_field work with an external file filled with values?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 9, 2016, 7:07pm UTC](https://discuss.elastic.co/t/delete-inner-fields-from-json-using-lookup-file/41294/2 "2016-02-09T19:07:14Z")

</div>

No, the mutate filter doesn't have such a feature. How often does this list of unwanted fields change? If it's not too often you could generate the necessary configuration. A few other options:

- You might be able to abuse the [translate filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html) to help out in some way (it can periodically reload a file from disk).
- The [ruby filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html) can definitely open and read files (but that'd be for each event, so it's costly).  
The [prune filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html) could also be useful.

---

<div class="post-metadata">

**Author:** ![Kryten](https://avatars.discourse-cdn.com/v4/letter/k/58956e/32.png) [@Kryten](https://discuss.elastic.co/u/Kryten)\
**Post date:** [February 10, 2016, 7:29am UTC](https://discuss.elastic.co/t/delete-inner-fields-from-json-using-lookup-file/41294/3 "2016-02-10T07:29:04Z")

</div>

Thanks Magnus,

I'll explore those options. The more I think about this I wonder about the viability of a new filter plugin that could be used to compare existing fields with a file containing known field names - where matching fields are then dropped or non-matching fields are dropped.

When parsing plain text logs I usually tag everything I want then drop anything that does not contain that tag to ensure sterile output, but when working with a json input ( codec=\>"json" ), it's extremely effective - but you get absolutely everything.

Cheers,  
Stuart

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:12am UTC](https://discuss.elastic.co/t/delete-inner-fields-from-json-using-lookup-file/41294/4 "2017-07-06T05:12:18Z")

</div>