# Flatten JSON Array in Logstash Filter

**URL:** <https://discuss.elastic.co/t/flatten-json-array-in-logstash-filter/124562>\
**Category:** Logstash\
**Created:** [March 19, 2018, 1:29pm UTC](https://discuss.elastic.co/t/flatten-json-array-in-logstash-filter/124562 "2018-03-19T13:29:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![EricPSU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericpsu/32/21110_2.png) [@EricPSU](https://discuss.elastic.co/u/EricPSU)\
**Post date:** [March 19, 2018, 1:29pm UTC](https://discuss.elastic.co/t/flatten-json-array-in-logstash-filter/124562/1 "2018-03-19T13:29:32Z")

</div>

I have a REST API call that returns the structure below. When I send this structure to Elasticsearch, all of the records within the "data" array are combined into one Elasticsearch document.

```
{
    "data": [
        {
            "updateDate": "2017/12/13 19:53",
            "id": "1234-ABCDE",
            "title": "Foo",
            "status": "ACTIVE",
            "key": {
                "number": 100,
                "version": 2,
                "year": 2017
            }
        },
        {
            "updateDate": "2017/12/14 10:22",
            "id": "4567-EFGHI",
            "title": "Bar",
            "status": "INACTIVE",
            "key": {
                "number": 200,
                "version": 5,
                "year": 2018
            }
        },
    ]
}

```

My goal is to manipulate the data to the following output so each "data" element is put into its own Elasticsearch document.

```
{
    "updateDate": "2017/12/13 19:53",
    "id": "1234-ABCDE",
    "title": "Foo",
    "status": "ACTIVE",
    "key": {
        "number": 100,
        "version": 2,
        "year": 2017
    }
},
{
    "updateDate": "2017/12/14 10:22",
    "id": "4567-EFGHI",
    "title": "Bar",
    "status": "INACTIVE",
    "key": {
        "number": 200,
        "version": 5,
        "year": 2018
    }
}

```

Is this possible in a filter? I've used the SPLIT plugin to break out each "data" element into its own document, but that results in fields that are still nested under a "data" element like this:

```
{
  "_index": "FOOBAR",
  "_type": "doc",
  "_id": "STdrPmIBPlj91gLKwYBv",
  "_score": 1,
  "_source": {
    "data": {
      "updateDate": "2017/12/13 19:53",
      "id": "1234-ABCDE",
      "title": "Foo",
      "status": "ACTIVE",
      "key": {
          "number": 100,
          "version": 2,
          "year": 2017
      }
    },
  "@version": "1",
  "@timestamp": "2018-03-19T13:21:11.794Z"
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 19, 2018, 1:33pm UTC](https://discuss.elastic.co/t/flatten-json-array-in-logstash-filter/124562/2 "2018-03-19T13:33:52Z")

</div>

You can use a mutate filter to move the fields to the top level of the document. If the field names aren't known beforehand you can use a ruby filter to iterate over them and move them all.

---

<div class="post-metadata">

**Author:** ![EricPSU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericpsu/32/21110_2.png) [@EricPSU](https://discuss.elastic.co/u/EricPSU)\
**Post date:** [March 19, 2018, 2:41pm UTC](https://discuss.elastic.co/t/flatten-json-array-in-logstash-filter/124562/3 "2018-03-19T14:41:17Z")

</div>

Thank you for the guidance! For the benefit of future readers, this is my working filter:

```
filter {

  split {     
    field => "[data]"
  }
  
  mutate {
    add_field => { 
      "id" => "%{[data][id]}"
      "updateDate" => "%{[data][updateDate]}"
      "title" => "%{[data][title]}"
      "status" => "%{[data][status]}"      
    }
    
    remove_field => ["[data]" ]
  }
  
}

```

It results in the structure below:

```
{
  "_index": "FOOBAR",
  "_type": "doc",
  "_id": "lDexPmIBPlj91gLKC7N7",
  "_score": 1,
  "_source": {
    "@timestamp": "2018-03-19T14:36:52.762Z",
    "updateDate": "2017/12/13 19:53",
    "title": "Foo",
    "status": "ACTIVE",
    "id": "1234-ABCDE",
    "@version": "1"
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 16, 2018, 2:41pm UTC](https://discuss.elastic.co/t/flatten-json-array-in-logstash-filter/124562/4 "2018-04-16T14:41:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.