# Limit of total fields \[1000\] in index exceeded and nested json

**URL:** <https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-exceeded-and-nested-json/247716>\
**Category:** Logstash\
**Created:** [September 7, 2020, 6:41am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-exceeded-and-nested-json/247716 "2020-09-07T06:41:51Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Luca\_P](https://avatars.discourse-cdn.com/v4/letter/l/bbe5ce/32.png) [@Luca\_P](https://discuss.elastic.co/u/Luca_P)\
**Post date:** [September 7, 2020, 6:41am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-exceeded-and-nested-json/247716/1 "2020-09-07T06:41:51Z")

</div>

Hi All, I need some help to avoid the well known 'Limit of total fields [1000] in index exceeded' problem. I don't consider to raise the default in the settings above 1000.  
The culprit seems to be a field with a json value that contains simple key-values as well as nested structures.  
The first problem is that messages with this field are shipped by different applications and only few keys inside that field are in common between all application. Other keys are unique to each application, and I don't know all of them. They even might change in the future, so hardcoding them in a mapping is not a practical option.  
Second problem I found, some of the keys inside that field have json object as value, some others have json strings, some others array of json.  
See a mock of what I see in the JSON tab in Kibana:

```auto
    {
      "_index": "my-index-01",
      "_type": "doc",
      "_id": "QJTIJXQBPFU8jmrx9tmo",
      "_version": 1,
      "_score": null,
      "_source": {
        "level": "info",
        "tags": [
          "winston"
        ],
    {
      "_index": "my-index-01",
      "_type": "doc",
      "_id": "QJTIJXQBPFU8jmrx9tmo",
      "_version": 1,
      "_score": null,
      "_source": {
        "level": "info",
        "tags": [
          "winston"
        ],
        "host": "ip-10-0-185-143.eu-west-1.compute.internal",
        "type": "tcp",
        . . .
        . . .
        "context": {
          "userType": "admin",
          "userDept": "admins",          
          "example": {
            "user": {
              "johndoe": "{\"name\":\"John\",\"surname\":\"Doe\"}"
              "isEmpty": false,
              "initDate": 1629806366409
            },
            "position": {
              "geo": {
                "x": "1234567890",
                "y": "0987654321"
              }
            },
            "textEntries": [
              {
                "key": "hasCredentials",
                "value": "false"
              },
              {
                "key": "isInternal",
                "value": "false"
              },
              {
                "key": "isMember",
                "value": "true"
              }
            ]
          }
        }
      }
    }

```

We use dynamic indexing, so I suspect that whenever one of those messages contains a "context" field that in turn contains very verbose nested values, we reach the 1000 limit and the message is not ingested.  
Is there a practical way to keep indexing the simple key-values inside "context" (like userType and userDepts), but avoid indexing the values of the keys when such values are strigified json or other nested structures ("example", "position", "textEntries", etc) ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 7, 2020, 4:05pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-exceeded-and-nested-json/247716/2 "2020-09-07T16:05:58Z")

</div>

If you want to end up with

```
   "context" => {
    "userDept" => "admins",
    "userType" => "admin"
},

```

then use

```
ruby {
    code => '
        ctx = event.get("context")
        if ctx
            event.set("context", ctx.delete_if { |k, v| v.is_a?(Hash) or v.is_a?(Array) } )
        end
    '
}

```

If you want to end up with

```
   "context" => {
    "userDept" => "admins",
    "userType" => "admin",
     "example" => "{\"position\"=>{\"geo\"=>{\"x\"=>\"1234567890\", \"y\"=>\"0987654321\"}}, \"textEntries\"=>[{\"value\"=>\"false\", \"key\"=>\"hasCredentials\"}, {\"value\"=>\"false\", \"key\"=>\"isInternal\"}, {\"value\"=>\"true\", \"key\"=>\"isMember\"}], \"user\"=>{\"isEmpty\"=>false, \"johndoe\"=>\"{\\\"name\\\":\\\"John\\\",\\\"surname\\\":\\\"Doe\\\"}\", \"initDate\"=>1629806366409}}"
},

```

then use

```
ruby {
    code => '
        ctx = event.get("context")
        if ctx
            ctx.each { |k, v|
                if v.is_a?(Hash) or v.is_a?(Array)
                    ctx[k] = v.to_s
                end
            }
            event.set("context", ctx)
        end
    '
}
```

---

<div class="post-metadata">

**Author:** ![Luca\_P](https://avatars.discourse-cdn.com/v4/letter/l/bbe5ce/32.png) [@Luca\_P](https://discuss.elastic.co/u/Luca_P)\
**Post date:** [September 8, 2020, 4:58am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-exceeded-and-nested-json/247716/3 "2020-09-08T04:58:38Z")

</div>

Thanks @Badger, the second option was what I was looking for !

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 6, 2020, 4:58am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-exceeded-and-nested-json/247716/4 "2020-10-06T04:58:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.