# Cannot manage nested fields

**URL:** <https://discuss.elastic.co/t/cannot-manage-nested-fields/88293>\
**Category:** Logstash\
**Created:** [June 5, 2017, 3:52pm UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293 "2017-06-05T15:52:42Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nikolay\_Petrov](https://avatars.discourse-cdn.com/v4/letter/n/4af34b/32.png) [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\
**Post date:** [June 5, 2017, 3:52pm UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/1 "2017-06-05T15:52:42Z")

</div>

Hi,  
could you please help to understand why removing of nested fields from 'grok' and 'mutate' is not working in my case, or how to remove them properly via filter settings, or any well know best practices regarding this are welcome as well!  
What is wrong in current configuration?

filter

> if [type] == "webapp\_access-events" {  
> grok {  
> patterns\_dir =\> ["/etc/logstash/patterns.d"]  
> match =\> { "message" =\> "%{GHTTP}" }  
> overwrite =\> ["message"]  
> #remove\_field =\> ["[webapp\_access-events][os]", "[webapp\_access-events][os\_name]", "[webapp\_access-events][beat.hostname]", "[webapp\_access-events][beat.version]", "[webapp\_access-events][input]" ]  
> }  
> mutate {  
> remove\_field =\> ["[webapp\_access-events][os]", "[webapp\_access-events][os\_name]", "[webapp\_access-events][beat.hostname]", "[webapp\_access-events][beat.version]", "[webapp\_access-events][input]" ]  
> }

> curl -s -XGET http://`hostname`:9200/webapp-events-2017.06.05/\_mapping/field/os\_name?pretty=true  
> {  
> "webapp-events-2017.06.05" : {  
> "mappings" : {  
> "webapp\_access-events" : {  
> "os\_name" : {  
> "full\_name" : "os\_name",  
> "mapping" : {  
> "os\_name" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> }  
> }

**logstash version 5.3.2**

> /usr/share/logstash/bin/logstash-plugin list | grep filter  
> ...  
> logstash-filter-grok  
> logstash-filter-mutate  
> ...

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 7, 2017, 6:41am UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/2 "2017-06-07T06:41:03Z")

</div>

Please show an example event from ES (or a `stdout { codec => rubydebug }` output).

---

<div class="post-metadata">

**Author:** ![Nikolay\_Petrov](https://avatars.discourse-cdn.com/v4/letter/n/4af34b/32.png) [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\
**Post date:** [June 7, 2017, 8:39pm UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/3 "2017-06-07T20:39:35Z")

</div>

Due to "Sorry, you can only mention 10 users in a post."..not sure what does it meant,but cutting the original message:

> {  
> "\_index": "webapp-events-2017.06.07",  
> "\_type": "webapp\_access-events",  
> "\_id": "AVyEHiFJrNmMuqrEW1OI",  
> "\_source": {  
> ...  
> "beat": {  
> "hostname": "hostname1",  
> "name": "hostname1",  
> "version": "5.4.0"  
> },  
> "host": "hostname1",  
> "client\_ip": "10.10.10.1",  
> "geoip": {},  
> "offset": 45834794,  
> "method": "GET",  
> "req\_file": "..reqfile..",  
> "os": "Other",  
> ...  
> "message": "..message..",  
> "tags": [  
> "beats\_input\_codec\_plain\_applied",  
> "\_geoip\_lookup\_failure"  
> ],  
> "referrer": ""-"",  
> "input": "735",  
> "@timestamp": "2017-06-07T19:53:00.000Z",  
> "response": 200,  
> "bytes": 498,  
> "name": "Other",  
> "os\_name": "Other",  
> "device": "Other"  
> }  
> ...

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 8, 2017, 5:05am UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/4 "2017-06-08T05:05:03Z")

</div>

Right, as I suspected. You say

```
remove_field => ["[webapp_access-events][os]", ...

```

as if `os` was a subfield of `webapp_access-events`, but it's actually a field at the root. So, turn

```
remove_field => ["[webapp_access-events][os]", "[webapp_access-events][os_name]", "[webapp_access-events][beat.hostname]", "[webapp_access-events][beat.version]", "[webapp_access-events][input]" ]

```

into this:

```
remove_field => ["os", "os_name", "[beat][hostname]", "[beat][version]", "input" ]

```

(Note `[beat][hostname]`, not `[beat.hostname]`.)

---

<div class="post-metadata">

**Author:** ![Nikolay\_Petrov](https://avatars.discourse-cdn.com/v4/letter/n/4af34b/32.png) [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\
**Post date:** [June 8, 2017, 11:48am UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/5 "2017-06-08T11:48:47Z")

</div>

Appreciate your help @magnusbaeck!  
Some of fields where successfully removed, some of them are not:

in conf.file:

> ...  
> grok {  
> ...  
> remove\_field =\> ["device", "name", "os", "os\_name", "output", "[beat][hostname]", "[beat][version]", "input", "input\_type", "ident", "version", "source", "host", "offset", "http\_version", "referrer" ]

but incoming fields looks like:

> ...  
> (metaFields)  
> ...  
> {  
> "\_source": {  
> "request": "req",  
> "agent": "agent",  
> "auth": "-",  
> "req\_param": "req\_param",  
> "type": "webapp\_access-events",  
> "serve\_in\_sec": "0",  
> "@version": "1",  
> "beat": {  
> "name": "hostname1"  
> },  
> "client\_ip": "10.10.10.1",  
> "geoip": {},  
> "method": "GET",  
> "req\_file": "reqfile",  
> "os": "Other",  
> "serve\_in\_msec": "296",  
> "message": "mess",  
> "tags": [  
> "beats\_input\_codec\_plain\_applied",  
> "\_geoip\_lookup\_failure"  
> ],  
> "@timestamp": "2017-06-08T11:36:58.000Z",  
> "response": 200,  
> "bytes": 498,  
> "name": "Other",  
> "os\_name": "Other",  
> "device": "Other"  
> },  
> "@timestamp": [  
> 1496921818000  
> ]  
> },  
> "sort": [  
> 1496921818000  
> ]  
> }  
> as you see for some reason part of them: "name", "os", "os\_name", "device" are still there.  
> Second question is 'remove\_filed' should be invoke from "grok" or by "mutate" ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 8, 2017, 12:03pm UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/6 "2017-06-08T12:03:04Z")

</div>

> as you see for some reason part of them: "name", "os", "os\_name", "device" are still there.

Please show your full configuration.

> Second question is 'remove\_filed' should be invoke from "grok" or by "mutate" ?

It depends. Normally, `remove_field` is only effective if the filter it's in was successful, i.e. if you put it in a grok filter it'll only remove fields if the grok filter matches. If it's the sole option in a mutate filter it'll run unconditionally.

---

<div class="post-metadata">

**Author:** ![Nikolay\_Petrov](https://avatars.discourse-cdn.com/v4/letter/n/4af34b/32.png) [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\
**Post date:** [June 8, 2017, 1:13pm UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/7 "2017-06-08T13:13:28Z")

</div>

cat conf.d/60-http.conf

> filter {  
> if [type] == "webapp\_access-events" {  
> grok {  
> patterns\_dir =\> ["/etc/logstash/patterns.d"]  
> match =\> { "message" =\> "%{GHTTP}" }  
> remove\_field =\> ["device", "name", "os", "os\_name", "output", "[beat][hostname]", "[beat][version]", "input", "input\_type", "ident", "version", "source", "host", "offset", "http\_version", "referrer" ]  
> overwrite =\> ["message"]  
> }
> 
> ```
> if "_grokparsefailure" in [tags] {
> drop {}
> }
> 
> mutate {
> convert => {"response" => "integer"}
> convert => {"bytes" => "integer"}
> }
> geoip {
> source => "client_ip"
> target => "geoip"
> #add_tag => ["apache-geoip"]
> }
> date {
> match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
> 
> remove_field => ["timestamp"]
> }
> useragent {
> source => "agent"
> }
> 
> ```
> 
> }  
> }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 8, 2017, 1:36pm UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/8 "2017-06-08T13:36:50Z")

</div>

The useragent filter produces the fields _after_ you've tried to remove them in your grok filter. Add a mutate filter that deletes them when they actually exist.

---

<div class="post-metadata">

**Author:** ![Nikolay\_Petrov](https://avatars.discourse-cdn.com/v4/letter/n/4af34b/32.png) [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\
**Post date:** [June 8, 2017, 2:02pm UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/9 "2017-06-08T14:02:47Z")

</div>

Awesome @magnusbaeck!  
It works!  
You are the best 😉

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:02pm UTC](https://discuss.elastic.co/t/cannot-manage-nested-fields/88293/10 "2017-07-06T14:02:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.