# Logstash Extract Elasticsearch Nested Fields

**URL:** <https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451>\
**Category:** Logstash\
**Created:** [June 9, 2021, 2:17pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451 "2021-06-09T14:17:28Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 9, 2021, 2:17pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/1 "2021-06-09T14:17:28Z")

</div>

Hello Team

I am using Elasticsearch version 7.8.0 and Logstash version 7.8.0

I am having ES Index having following mappings :

```auto
    "mappings" : {
      "properties" : {
        "appEvents" : {
          "properties" : {
            "attribute" : {
              "properties" : {
                "filesPendingForUpload" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "filesUploadedSinceLastEvent" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "launch" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },

```

Now i want to extract the data from ES Index to CSV file using logstash.

In logstash conf file I have specified

```auto
output {
  csv {
    fields => ["[appEvents][attribute][filesPendingForUpload]"]
    path => "/tmp/exp.csv"
  }
}

```

But I am not getting respective field data in csv.

As per my understanding, I am not writing fields in proper format...

Could you please help me in getting data into csv.

Thank You.

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 10, 2021, 5:44am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/2 "2021-06-10T05:44:18Z")

</div>

I meant to ask that .. how to mention fields having multi-nested ones in logstash ...

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 11, 2021, 5:41am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/3 "2021-06-11T05:41:30Z")

</div>

Hello Team

I tried using filter = mutate = for merging multiple fields into single field ...

```auto
filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][attribute_num][filesUploadedSinceLastEvent]}" }
  }
}

```

But my output contains :

```auto
6018a66855b6471ebf976b2d551c9731,1623241661046,Y,WLC,Y,Y,CAM,Original,,OFF,%{[appEvents][attribute_num][filesPendingForUpload]},%{[appEvents][attribute_num][filesUploadedSinceLastEvent]}

```

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 11, 2021, 9:53am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/4 "2021-06-11T09:53:11Z")

</div>

Below data i want to extract using logstash into CSV file ...

I tried multiple ways , but not able to achieve it...

Could someone please help me ...

````auto
"appEvents":[{"attribute_num":{"filesPendingForUpload":8,"filesUploadedSinceLastEvent":173}}]```
````

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 11, 2021, 11:34am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/5 "2021-06-11T11:34:59Z")

</div>

can someone please help me how to mentioned deep nested fields in logstash

output plugin of csv : FIELDS

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 14, 2021, 2:46am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/6 "2021-06-14T02:46:25Z")

</div>

Can someone please help me ...

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 14, 2021, 3:18am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/7 "2021-06-14T03:18:41Z")

</div>

> [@tusharnemade](#):
>
> Can someone please help me ...

Not easily. You have shown that you have an elasticsearch mapping. That suggests that you have an index somewhere. It gives no indication of what kind of events the logstash input is producing.

The output plugin configuration you show would produce a single column CSV, but you say your output contains multiple columns.

sprintf references like `%{[appEvents][attribute_num][filesUploadedSinceLastEvent]}` showing up in the output would be consistent with that field not existing and you referencing [filesUploadedSinceLastEvent] in the csv filter after an event went through that mutate filter.

Basically it is impossible for what you describe to be happening, and we cannot guess which parts of your description are wrong.

We are happy to help, but you have to help us to help you.

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 14, 2021, 3:25am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/8 "2021-06-14T03:25:18Z")

</div>

Hello Badger

Thanks for your response.

Let me put it again properly ...

my logstash conf file is below one :

My Input if Elasticsearch Index version 7.8.0

```auto
input {
  elasticsearch {
    hosts => "10.10.10.10:9200"
    query => '{"_source" : ["userId", "timeStamp", "backupSettings.backgroundUploading", "backupSettings.connectionType", "backupSettings.contactBackup", "backupSettings.contactPermission", "backupSettings.photoBackup", "backupSettings.photoQuality", "backupSettings]storagePermission", "backupSettings.videoBackup", "appEvents.attribute_num.filesPendingForUpload","appEvents.attribute_num.filesUploadedSinceLastEvent"],"query" : { "match_all": {} }}'
    size => 10000
    scroll => "5m"
    index => "backupindex"
  }
}

```

Now I have filter added to take multi-nested fields into single new field.

```auto
filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[[appEvents][attribute_num]][filesPendingForUpload]}" }
                add_field => { "filesUploadedSinceLastEvent" => "%{[[appEvents][attribute_num]][filesUploadedSinceLastEvent]}" }
    }
}

```

Now output goes to CSV File :

```auto
output {
  csv {
    fields => ["userId", "timeStamp", "[backupSettings][backgroundUploading]", "[backupSettings][connectionType]", "[backupSettings][contactBackup]", "[backupSettings][contactPermission]", "[backupSettings][photoBackup]", "[backupSettings][photoQuality]", "[backupSettings][storagePermission]", "[backupSettings][videoBackup]", "[filesPendingForUpload]", "[filesUploadedSinceLastEvent]"]
    path => "/tmp/exp.csv"
  }
}

```

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 14, 2021, 5:21am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/9 "2021-06-14T05:21:29Z")

</div>

Still the implementation is unclear from my side in above explanation ?

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 14, 2021, 10:03am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/10 "2021-06-14T10:03:11Z")

</div>

Hi

Could some one please help me ..

IF still my description of my issue is incomplete , please do let me know .. I will fill up the gap...

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 14, 2021, 1:47pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/11 "2021-06-14T13:47:49Z")

</div>

> [@tusharnemade](#):
>
> ```auto
> add_field => { "filesPendingForUpload" => "%{[[appEvents][attribute_num]][filesPendingForUpload]}" }
> add_field => { "filesUploadedSinceLastEvent" => "%{[[appEvents][attribute_num]][filesUploadedSinceLastEvent]}" }
> 
> ```

The output suggests that these are not working. Why do you have a second set of square brackets around [appEvents][attribute\_num]?

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 14, 2021, 1:53pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/12 "2021-06-14T13:53:54Z")

</div>

Hi Badger

I tried with below options :

1. 

```auto
add_field => { "filesPendingForUpload" => "%{[appEvents][attribute_num][filesPendingForUpload]}" }
                add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][attribute_num][filesUploadedSinceLastEvent]}" }

```

1. 

```auto
add_field => { "filesPendingForUpload" => "%{[[appEvents][attribute_num]][filesPendingForUpload]}" }
                add_field => { "filesUploadedSinceLastEvent" => "%{[[appEvents][attribute_num]][filesUploadedSinceLastEvent]}" }

```

1. 

```auto
add_field => { "filesPendingForUpload" => "%{[appEvents][[attribute_num][filesPendingForUpload]]}" }
                add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][[attribute_num][filesUploadedSinceLastEvent]]}" }

```

I came across some forums : for multi nested path use [] double square brackets ...

I tried above all 3 options , but none worked...

Could you please help me to achieve data of , below fields in logstash CSV output plugin. How to mention such fields ..

```auto
appEvents.attribute_num.filesPendingForUpload 
appEvents.attribute_num.filesUploadedSinceLastEvent

```

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 14, 2021, 4:52pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/13 "2021-06-14T16:52:31Z")

</div>

I also tried below :

1. specifying ` "[appEvents][0][attribute_num][filesPendingForUpload]"`

```auto
input {
  elasticsearch {
    hosts => "10.10.101.1:9200"
    size => 10000
    index => "backupindex"
  }
}

output {
  csv {
    fields => ["userId", "timeStamp", "[backupSettings][backgroundUploading]", "[backupSettings][connectionType]", "[backupSettings][contactBackup]", "[backupSettings][contactPermission]", "[backupSettings][photoBackup]", "[backupSettings][photoQuality]", "[backupSettings][storagePermission]", "[backupSettings][videoBackup]", "[appEvents][0][attribute_num][filesPendingForUpload]"]
    path => "/tmp/exp.csv"
  }
}

```

1. specifying ` "[appEvents][attribute_num][filesPendingForUpload]"`

Both did not helped me ... IN CSV , field was blank.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 14, 2021, 4:54pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/14 "2021-06-14T16:54:25Z")

</div>

Can you replace your output with

output { stdout { codec =\> rubydebug } }

and show us what [appEvents] looks like?

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 15, 2021, 3:57am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/15 "2021-06-15T03:57:13Z")

</div>

Okay Sure ..

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 15, 2021, 5:05am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/16 "2021-06-15T05:05:41Z")

</div>

> [@Badger](#):
>
> output { stdout { codec =\> rubydebug } }

```auto
{
         "timeStamp" => 1623203744873,
        "@timestamp" => 2021-06-15T04:55:11.613Z,
          "@version" => "1",
            "userId" => "c0135af5feb34a77baee92b7551c4d06",
         "appEvents" => [
        [0] {
            "attribute_num" => {
                "filesUploadedSinceLastEvent" => 0,
                      "filesPendingForUpload" => 2
            }
        }
    ],
    "backupSettings" => {
             "connectionType" => "WLC",
        "backgroundUploading" => "Y",
          "contactPermission" => "Y",
                "photoBackup" => "CAM",
               "photoQuality" => "Original",
                "videoBackup" => "OFF",
              "contactBackup" => "Y",
          "storagePermission" => "Y"
    }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 15, 2021, 2:22pm UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/17 "2021-06-15T14:22:41Z")

</div>

> [@tusharnemade](#):
>
> ```auto
> "appEvents" => [
> [0] {
> "attribute_num" => {
> 
> ```

[appEvents] is an array, so you should be using "[appEvents][0][attribute\_num][filesPendingForUpload]"

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 2:41am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/18 "2021-06-16T02:41:35Z")

</div>

Hello Badger

I have already tried using this ...

Could you please have a look over this post ..

> [@Logstash Extract Elasticsearch Nested Fields](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/13):
>
> I also tried below : specifying "[appEvents][0][attribute\_num][filesPendingForUpload]" input { elasticsearch { hosts =\> "10.10.101.1:9200" size =\> 10000 index =\> "backupindex" } } output { csv { fields =\> ["userId", "timeStamp", "[backupSettings][backgroundUploading]", "[backupSettings][connectionType]", "[backupSettings][contactBackup]", "[backupSettings][contactPermission]", "[backupSettings][photoBackup]", "[backupSettings][photoQuality]", "[backupSettings][stora…

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 16, 2021, 3:04am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/19 "2021-06-16T03:04:20Z")

</div>

I do not know what to say. Both the mutate+add\_field and the csv output will event.get "[appEvents][0][attribute\_num][filesPendingForUpload]". If that field exists it should be expanded.

Are you getting this for every event?

---

<div class="post-metadata">

**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\
**Post date:** [June 16, 2021, 3:11am UTC](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451/20 "2021-06-16T03:11:28Z")

</div>

Hi Badger

I did not used mutate this time ...

```auto
input {
  elasticsearch {
    hosts => "10.10.101.10:9200"
    size => 10000
    index => "backup"
  }
}

output {
  csv {
    fields => ["userId", "timeStamp", "[backupSettings][backgroundUploading]", "[backupSettings][connectionType]", "[backupSettings][contactBackup]", "[backupSettings][contactPermission]", "[backupSettings][photoBackup]", "[backupSettings][photoQuality]", "[backupSettings][storagePermission]", "[backupSettings][videoBackup]", "[appEvents][0][attribute_num][filesUploadedSinceLastEvent]", "[appEvents][0][attribute_num][filesPendingForUpload]"]
    path => "/tmp/exp.csv"
  }
}

```

Output :

```auto
6018a66855b6471ebf976b2d551c9731,1623241661046,Y,WLC,Y,Y,CAM,Original,Y,OFF,,

```

You can see last both fields are empty ...

they should contain data

[Next page](https://discuss.elastic.co/t/logstash-extract-elasticsearch-nested-fields/275451.md?page=2)