Indexing documents with nested fields

[Nikhil_sharma1 (Nikhil sharma)]

Hi,

We have setup ELK infra on AWS. I would like to understand is there any limitation from ELK side to the numbers of fields that can be indexed? As, our schema has a Map fields and it has many unique fields in it, the index size is growing and slowing down ELK. Do you have recommendations around that can be done to optimize it? It has been communicated by our infra team that ELK has a limitation of indexing only 1000 unique fields (is it true?)

[spinscale (Alexander Reelsen)]

There is indeed a limit on the number of fields. Instead of having many different fields maybe you can create a common schema across all your documents - or index these documents in to different indices instead, where you can have of course differently named fields (again up to 1k).

You may want to take a look at the elastic common schema for some inspiration.

See more at https://www.elastic.co/blog/brewing-in-beats-introduce-elastic-common-schema

--Alex

[Nikhil_sharma1 (Nikhil sharma)]

Hi Alexander,
Thanks for your reply! How to optimize the scenario where I have a nested child json and that json has no schema attached to it! Basically the users can put in any fields that they want in that JSON (its a HashMap<String,String>). Is there any recommendation for such kind of data?

[RahulD (Rahul Desai)]

You can store the key part of the map inside a nested field called key_field and the value part of the map in a field called value_field. Both key_field and value_field can have type = keyword. That way the nested document only ever has 2 fields that will save you from your index mapping growing over time.