# How to remove fields with regex from json?

**URL:** <https://discuss.elastic.co/t/how-to-remove-fields-with-regex-from-json/111776>\
**Category:** Logstash\
**Created:** [December 14, 2017, 1:13pm UTC](https://discuss.elastic.co/t/how-to-remove-fields-with-regex-from-json/111776 "2017-12-14T13:13:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/son/32/25572_2.png) [@son](https://discuss.elastic.co/u/son)\
**Post date:** [December 14, 2017, 1:13pm UTC](https://discuss.elastic.co/t/how-to-remove-fields-with-regex-from-json/111776/1 "2017-12-14T13:13:09Z")

</div>

Hi,

I am stucking in removing fields with regex. I have log in json. From filebeat i have configured it to push to logstash via config:

> filebeat.prospectors:
> 
> - input\_type: log  
> paths:
> - E:\test\app\Logging\test.json  
> json.keys\_under\_root: true  
> json.overwrite\_keys: false  
> json.add\_error\_key: true

The log content will be something like this:

> {"hi":"hello","test":{"qs: abc":"1","qs:\_12313":"2", "demo":"test"}}

nested object "qs" is random changed and I want to remove them, just keep "demo" and "hi".

Could you advise me how to remove it please?

Thanks in advance.

* * *

Son

---

<div class="post-metadata">

**Author:** ![son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/son/32/25572_2.png) [@son](https://discuss.elastic.co/u/son)\
**Post date:** [December 16, 2017, 2:59am UTC](https://discuss.elastic.co/t/how-to-remove-fields-with-regex-from-json/111776/2 "2017-12-16T02:59:45Z")

</div>

Can somebody help me please?

I am trying with below ruby config but i don't know how to remove subfield due to the eventapi has changed from version 5.x:

> ruby {  
> code =\> "  
> if event.get('test') != nil  
> event.get('test').to\_hash.keys.each { |k|  
> if k =~ /qs\*/  
> event.remove(k)  
> end  
> }  
> end  
> "  
> }  
> }

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![tatdat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tatdat/32/113160_2.png) [@tatdat](https://discuss.elastic.co/u/tatdat)\
**Post date:** [December 17, 2017, 5:45am UTC](https://discuss.elastic.co/t/how-to-remove-fields-with-regex-from-json/111776/3 "2017-12-17T05:45:00Z")

</div>

Do you try with mutate plugin?

```
muate {
   remove_field => ["test.qs"]   
}
```

---

<div class="post-metadata">

**Author:** ![son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/son/32/25572_2.png) [@son](https://discuss.elastic.co/u/son)\
**Post date:** [December 17, 2017, 7:23am UTC](https://discuss.elastic.co/t/how-to-remove-fields-with-regex-from-json/111776/4 "2017-12-17T07:23:00Z")

</div>

Hi,

"qs" is just prefix, there are random strings trailing (qs: abc, qs:\_demo, qs: 123 and more). That's why i don't know how to remove with regex. I gave it some tries with other plugin but no luck.

Could you advise please?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![cb2015](https://avatars.discourse-cdn.com/v4/letter/c/7993a0/32.png) [@cb2015](https://discuss.elastic.co/u/cb2015)\
**Post date:** [December 18, 2017, 10:47pm UTC](https://discuss.elastic.co/t/how-to-remove-fields-with-regex-from-json/111776/5 "2017-12-18T22:47:48Z")

</div>

Ive never tried this on nested objects, but when I ingest log files I filter out noise(pings, etc) by using

```
        if [message] =~ "blah" {
                    grok {
            match => { "message" => "%{BASE10NUM:Stored_Procedure_Exec_Time_Ms}" }
        }
    }

```

I used the grok filter here, but you should look into the drop filter. Just as a reminder, when you deal with nested objects in logstash, you should use [field][nestedField] as the format.

---

<div class="post-metadata">

**Author:** ![son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/son/32/25572_2.png) [@son](https://discuss.elastic.co/u/son)\
**Post date:** [December 19, 2017, 8:48am UTC](https://discuss.elastic.co/t/how-to-remove-fields-with-regex-from-json/111776/6 "2017-12-19T08:48:45Z")

</div>

Hi Cody,

There is no regex for nested object as i checked. That's why i had to use ruby filter. But unfortunately i don't know ruby so it is my issue now. I cannot remove nested field with ruby code.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 16, 2018, 8:48am UTC](https://discuss.elastic.co/t/how-to-remove-fields-with-regex-from-json/111776/7 "2018-01-16T08:48:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.