# Logstash Extract Elasticsearch Nested Fields

**URL:** <https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451>\
**Category:** Logstash\
**Created:** [June 9, 2021, 2:17pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451 "2021-06-09T14:17:28Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 3:25am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/21 "2021-06-16T03:25:08Z")

</div>

Could you please help me with what I am missing because of which my fields are not getting captured in csv file..

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 3:32am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/22 "2021-06-16T03:32:38Z")

</div>

There is one hope now ....

I used mutate fields ... and i got the record successfully.

Only problem is now if event is empty .. then it prints entire mutate field.

Could you please suggest ways to avoid this...

```auto
input {
  elasticsearch {
    hosts => "10.10.100.10:9200"
    query => '{"_source" : ["userId", "timeStamp", "backupSettings.backgroundUploading", "backupSettings.connectionType", "backupSettings.contactBackup", "backupSettings.contactPermission", "backupSettings.photoBackup", "backupSettings.photoQuality", "backupSettings.storagePermission", "backupSettings.videoBackup", "appEvents.attribute_num.filesPendingForUpload","appEvents.attribute_num.filesUploadedSinceLastEvent"],"query" : { "match_all": {} }}'
    size => 10000
    scroll => "5m"
    index => "backup"
  }
}

filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
}

output {
  csv {
    fields => ["userId", "timeStamp", "[backupSettings][backgroundUploading]", "[backupSettings][connectionType]", "[backupSettings][contactBackup]", "[backupSettings][contactPermission]", "[backupSettings][photoBackup]", "[backupSettings][photoQuality]", "[backupSettings][storagePermission]", "[backupSettings][videoBackup]", "filesPendingForUpload", "filesUploadedSinceLastEvent"]
    path => "/tmp/exp.csv"
  }
}

```

```auto
6018a66855b6471ebf976b2d551c9731,1623241661046,Y,WLC,Y,Y,CAM,Original,Y,OFF,8,173
6018a66855b6471ebf976b2d551c9731,1623256154896,,,,,,,,,%{[appEvents][0][attribute_num][filesPendingForUpload]},%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}
6018a66855b6471ebf976b2d551c9731,1623254680762,,,,,,,,,%{[appEvents][0][attribute_num][filesPendingForUpload]},%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 16, 2021, 3:44am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/23 "2021-06-16T03:44:46Z")

</div>

Possibly use a prune filter where the [default](https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html#plugins-filters-prune-blacklist_names) value for blacklist\_names will delete fields where a sprintf reference failed. I think the unresolved sprintf reference will be top-level so then you should get an empty field.

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 3:49am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/24 "2021-06-16T03:49:46Z")

</div>

Thank You for your response.

I will check and reply in here ...

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 11:30am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/25 "2021-06-16T11:30:12Z")

</div>

Hello Badger

I tried below ways .. but it did not helped me ...  
I am sure , my writing of prune filter is not correct .. .hence its not working properly as expected ..

Could you please help me with its syntax to be mentioned ...

1. 

```auto
filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
    prune {
        blacklist_names => ["filesPendingForUpload", "filesUploadedSinceLastEvent"]
    }
}

```

1. 

```auto
filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
    prune {
        blacklist_names => ["filesPendingForUpload", "filesUploadedSinceLastEvent"]
        blacklist_values => ["" , ""]
    }
}

```

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 1:35pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/26 "2021-06-16T13:35:43Z")

</div>

I also tried below :

```auto
filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
    prune {
        blacklist_values => ["filesPendingForUpload" ,"" , "filesUploadedSinceLastEvent", ""]
    }
}

```

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 1:58pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/27 "2021-06-16T13:58:42Z")

</div>

I would like to high light in here that for some documents we have this field missing ....

for example below is the set of document does not have  
`[appEvents][0][attribute_num][filesUploadedSinceLastEvent]` and` [appEvents][0][attribute_num][filesPendingForUpload]`

```auto
{
        "userId" => "6018a66855b6471ebf976b2d551c9731",
    "@timestamp" => 2021-06-16T13:40:32.584Z,
      "@version" => "1",
     "timeStamp" => 1623256154896
}

```

In such cases , output is like below in csv file

```auto
6018a66855b6471ebf976b2d551c9731,1623256154896,,,,,,,,,%{[appEvents][0][attribute_num][filesPendingForUpload]},%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}
6018a66855b6471ebf976b2d551c9731,1623254680762,,,,,,,,,%{[appEvents][0][attribute_num][filesPendingForUpload]},%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}

```

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 2:07pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/28 "2021-06-16T14:07:02Z")

</div>

Finally I have achieved it ...

```auto
filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
    if ![appEvents][0][attribute_num][filesPendingForUpload] {
        mutate { remove_field => ["filesPendingForUpload"] }
}
    if ![appEvents][0][attribute_num][filesUploadedSinceLastEvent] {
        mutate { remove_field => ["filesUploadedSinceLastEvent"] }
}
}

```

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 2:13pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/29 "2021-06-16T14:13:37Z")

</div>

Thank You very much Badger for your responses ..

1. You mentioned that mutate + add\_filed should have helped me ...

> [@Logstash Extract Elasticsearch Nested Fields](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/19):
>
> I do not know what to say. Both the mutate+add\_field and the csv output will event.get "[appEvents][0][attribute\_num][filesPendingForUpload]". If that field exists it should be expanded. Are you getting this for every event?

Then i added my mutate code back to config file and it was giving results.

1. You mentioned to use prune filter ...

> [@Logstash Extract Elasticsearch Nested Fields](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/23):
>
> Possibly use a prune filter where the [default](https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html#plugins-filters-prune-blacklist_names) value for blacklist\_names will delete fields where a sprintf reference failed. I think the unresolved sprintf reference will be top-level so then you should get an empty field.

Although it did not helped me .. but I got the direction to think in the manner of fields and I used if conditions and finally i got the expected output ...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 14, 2021, 2:14pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/30 "2021-07-14T14:14:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451.md?page=1)