# Parsing nested json

**URL:** <https://discuss.elastic.co/t/parsing-nested-json/374629>\
**Category:** Elasticsearch\
**Created:** [February 17, 2025, 12:59pm UTC](https://discuss.elastic.co/t/parsing-nested-json/374629 "2025-02-17T12:59:58Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![user-27022024](https://avatars.discourse-cdn.com/v4/letter/u/ce7236/32.png) [@user-27022024](https://discuss.elastic.co/u/user-27022024)\
**Post date:** [February 17, 2025, 12:59pm UTC](https://discuss.elastic.co/t/parsing-nested-json/374629/1 "2025-02-17T12:59:58Z")

</div>

I am having issues with our `AWS ecs` -\> `fluentbit` -\> `elasticsearch` set up, specifically around nested json.

For example, if the log message is:

```auto
{
  "endpoint": "/process",
  "payload": {
     "body": {
       "success": "true",
       "items": [
          {"name": "item_one"},
          {"name": "item_two"}
       ]   
     }
  }
}

```

We would like the following fields to be parsed:

`endpoint` -\> `"process"`  
`payload` -\> `{"body": {"success": "true", "items": {"name": "item_one"}, {"name": "item_two"}]}`

Only the top level key.

We set `"index.mapping.depth.limit": 1` but this resulted in the logs being rejected by elasticsearch

```auto
    "status":400,
    "error":{
        "type": "illegal_argument_exception",
        "reason": "Limit of mapping depth [1] has been exceeded due to object field [org]"
    }

```

Is there a setting that will parse only the top level but accept the rest of the data as the body?

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [February 17, 2025, 1:30pm UTC](https://discuss.elastic.co/t/parsing-nested-json/374629/2 "2025-02-17T13:30:33Z")

</div>

Hey,

I'll try my best but you might want to wait for better answers 😃

I do believe it's a mapping issue, here elestic is rejecting because the setting is not defined ?

You could also try to set the fields a keyword but it would be a static per field definition which can be tedious in case you have mixed data with a lot of fields.

Also there is a lot of ruby scripts around here if you want to extract the subkeys.

---

<div class="post-metadata">

**Author:** ![user-27022024](https://avatars.discourse-cdn.com/v4/letter/u/ce7236/32.png) [@user-27022024](https://discuss.elastic.co/u/user-27022024)\
**Post date:** [February 17, 2025, 2:35pm UTC](https://discuss.elastic.co/t/parsing-nested-json/374629/3 "2025-02-17T14:35:34Z")

</div>

> [@grumo35](#):
>
> believe it's a mapping issue, here elestic is rejecting because the setting is not defined ?

I appreciate the help!

So you are saying, I should try get the parsing done correctly on the fluentbit level and not elasticsearch level?

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [February 17, 2025, 3:27pm UTC](https://discuss.elastic.co/t/parsing-nested-json/374629/4 "2025-02-17T15:27:29Z")

</div>

No you'll have to define a static type for the incoming field on the elastic side so that the indexed field values are of type text [Field data types | Elasticsearch Guide [8.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)

---

<div class="post-metadata">

**Author:** ![user-27022024](https://avatars.discourse-cdn.com/v4/letter/u/ce7236/32.png) [@user-27022024](https://discuss.elastic.co/u/user-27022024)\
**Post date:** [February 18, 2025, 1:53pm UTC](https://discuss.elastic.co/t/parsing-nested-json/374629/5 "2025-02-18T13:53:43Z")

</div>

Got it thanks.

We were hoping of not having to maintain the mappings on the elastic level but perhaps we'll have to.

Would it be possible to parse it at the fluentbit level so that the second level is a string instead of a object? Just trying to work out if there is a best practise for us to follow

---

<div class="post-metadata">

**Author:** ![user-27022024](https://avatars.discourse-cdn.com/v4/letter/u/ce7236/32.png) [@user-27022024](https://discuss.elastic.co/u/user-27022024)\
**Post date:** [February 20, 2025, 2:20pm UTC](https://discuss.elastic.co/t/parsing-nested-json/374629/6 "2025-02-20T14:20:14Z")

</div>

In the end we sorted it out at the pipeline level

```auto
PUT /_ingest/pipeline/main_pipeline
{
  
  "description" : "process-pipeline ",
  "processors" : [
      {
        "date" : {
          "field" : "timestamp",
          "formats" : ["ISO8601"],
          "ignore_failure" : true
        }
      }, 
      {
      "script": {
        "source": """
          for (entry in ctx.entrySet()) {
            if (entry.getValue() instanceof Map) {
              ctx[entry.getKey()] = entry.getValue().toString();
            }
          }
        """
      }
    }
  ]
}

```

Not sure if it's best practise but it worked

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [February 20, 2025, 2:55pm UTC](https://discuss.elastic.co/t/parsing-nested-json/374629/7 "2025-02-20T14:55:43Z")

</div>

Yes ! That's the script i was thinking about you solved that from the parsing side.

I suggest you also check data types of your mapping ( which i believe is automatic ) So that you understand the type must match and can be conflicting or sometimes prevent the document from being ingested correctly by elastic.