# Logstash : How to extract a nested field from Json log and only index the content of the nested field

**URL:** <https://discuss.elastic.co/t/logstash-how-to-extract-a-nested-field-from-json-log-and-only-index-the-content-of-the-nested-field/317617>\
**Category:** Logstash\
**Created:** [October 27, 2022, 1:11pm UTC](https://discuss.elastic.co/t/logstash-how-to-extract-a-nested-field-from-json-log-and-only-index-the-content-of-the-nested-field/317617 "2022-10-27T13:11:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ranjith\_kk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjith_kk/32/40809_2.png) [@Ranjith\_kk](https://discuss.elastic.co/u/Ranjith_kk)\
**Post date:** [October 27, 2022, 1:11pm UTC](https://discuss.elastic.co/t/logstash-how-to-extract-a-nested-field-from-json-log-and-only-index-the-content-of-the-nested-field/317617/1 "2022-10-27T13:11:29Z")

</div>

We have some logs in JSON format with a nested field called "data". We are looking for an option to extract only the content of this nested field and send it for indexing with ES.

Actual log format:

```auto
{"field1":"value1", "field2":"value2", "field3":"value3", "field4":"value4", "data":{"nested_field1":"nested_value1","nested_field2":"nested_value2", "nested_field3":"nested_value3"}}

```

Logs need to be sent to ES:

```auto
{"nested_field1":"nested_value1","nested_field2":"nested_value2", "nested_field3":"nested_value3"}

```

I was trying to use the Logstash Config below. But this does not work:

```auto
input {
 file {
   type => "json"
   path => "/home/ranjith/logstash.log"
   start_position => beginning
   sincedb_path => "/dev/null"
 }
}
filter {
      json {
        source => "message"
      }

      mutate {
        add_field => {"data" => "%{[message][data]}"}
        remove_field => "message"
        }
}
output {
stdout { codec => json }
} 

```

Any help is appreictaed.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 27, 2022, 1:23pm UTC](https://discuss.elastic.co/t/logstash-how-to-extract-a-nested-field-from-json-log-and-only-index-the-content-of-the-nested-field/317617/2 "2022-10-27T13:23:52Z")

</div>

What is your output? You need to share the output you are getting.

Also, since you used the `json` to parse your `message` field, your fields will be in the root of the event, so you will have a `data` field, not a `message.data` field, you basically do not need that `mutate` filter as you already have the `data` field.

If you want to limit the fields you will send to elasticsearch you will need to use the [prune](https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html) filter.

---

<div class="post-metadata">

**Author:** ![Ranjith\_kk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjith_kk/32/40809_2.png) [@Ranjith\_kk](https://discuss.elastic.co/u/Ranjith_kk)\
**Post date:** [October 27, 2022, 5:23pm UTC](https://discuss.elastic.co/t/logstash-how-to-extract-a-nested-field-from-json-log-and-only-index-the-content-of-the-nested-field/317617/3 "2022-10-27T17:23:30Z")

</div>

Hello leandrojmp,

Thanks for your support on this. I was able to make some progress with the help of your suggestions. With the new logstash config, I was able to extract only data field, but still encapsulating all the other fields. I want to take all the fields outside the "data" nest

Input message:  
`{"field1":"value1", "field2":"value2", "field3":"value3", "field4":"value4", "data":{"nested_field1":"nested_value1","nested_field2":"nested_value2", "nested_field3":"nested_value3"}}`

Current outpout with the Logstash config below:  
`{"data":{"nested_field3":"nested_value3","nested_field1":"nested_value1","nested_field2":"nested_value2"}}`

Expected output:  
`{"nested_field3":"nested_value3","nested_field1":"nested_value1","nested_field2":"nested_value2"}`  
We would not be able to use static field names as the fields under data{} can be dynamic. I would need something like [data][\*]

New logstash config:

```auto
input {
 file {
   type => "json"
   path => "/home/ranjith/logstash1.log"
   start_position => beginning
   sincedb_path => "/dev/null"
 }
}
filter {
      json {
        source => "message"
      }
      prune {
        whitelist_names => ["data"]
      }

      mutate {
        remove_field => ["message"]
      }
}
output {
stdout { codec => json }
}

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 27, 2022, 5:34pm UTC](https://discuss.elastic.co/t/logstash-how-to-extract-a-nested-field-from-json-log-and-only-index-the-content-of-the-nested-field/317617/4 "2022-10-27T17:34:33Z")

</div>

I think that you will need to use the ruby filter to put the nested fields under data into the root level of the document.

I'm not an expert in ruby, but this [other question](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/2) has an example that may work in your case.

It would be somehint like this, but you will need to test it out.

```auto
ruby { 
    code => 'event.get("data").each { | k, v| event.set(k, v) }' 
}
mutate { 
    remove_field => ["data"] 
}

```

Those filters would need to be after the `prune` filter.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 27, 2022, 5:58pm UTC](https://discuss.elastic.co/t/logstash-how-to-extract-a-nested-field-from-json-log-and-only-index-the-content-of-the-nested-field/317617/5 "2022-10-27T17:58:40Z")

</div>

> [@leandrojmp](#):
>
> ```auto
> ruby { 
> code => 'event.get("data").each { | k, v| event.set(k, v) }' 
> }
> mutate { 
> remove_field => ["data"] 
> }
> 
> ```

You can reduce that to

```
ruby { 
    code => 'event.remove("data").each { | k, v| event.set(k, v) }' 
}

```

---

<div class="post-metadata">

**Author:** ![Ranjith\_kk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjith_kk/32/40809_2.png) [@Ranjith\_kk](https://discuss.elastic.co/u/Ranjith_kk)\
**Post date:** [October 27, 2022, 6:38pm UTC](https://discuss.elastic.co/t/logstash-how-to-extract-a-nested-field-from-json-log-and-only-index-the-content-of-the-nested-field/317617/6 "2022-10-27T18:38:57Z")

</div>

That worked.. You are a saviour... Thank you for all your support.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 24, 2022, 6:39pm UTC](https://discuss.elastic.co/t/logstash-how-to-extract-a-nested-field-from-json-log-and-only-index-the-content-of-the-nested-field/317617/7 "2022-11-24T18:39:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.