Indexing documents with nested fields

Elastic Stack Elasticsearch
1

Hi,

We have setup ELK infra on AWS. I would like to understand is there any limitation from ELK side to the numbers of fields that can be indexed? As, our schema has a Map fields and it has many unique fields in it, the index size is growing and slowing down ELK. Do you have recommendations around that can be done to optimize it? It has been communicated by our infra team that ELK has a limitation of indexing only 1000 unique fields (is it true?)

2

There is indeed a limit on the number of fields. Instead of having many different fields maybe you can create a common schema across all your documents - or index these documents in to different indices instead, where you can have of course differently named fields (again up to 1k).

You may want to take a look at the elastic common schema for some inspiration.

See more at https://www.elastic.co/blog/brewing-in-beats-introduce-elastic-common-schema

--Alex

3

Hi Alexander,
Thanks for your reply! How to optimize the scenario where I have a nested child json and that json has no schema attached to it! Basically the users can put in any fields that they want in that JSON (its a HashMap<String,String>). Is there any recommendation for such kind of data?

4

You can store the key part of the map inside a nested field called key_field and the value part of the map in a field called value_field. Both key_field and value_field can have type = keyword. That way the nested document only ever has 2 fields that will save you from your index mapping growing over time.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.