Understanding field limit across index pattern and solution to mapping explosion

[elk11]

Hi,

elasticsearch and kibana version: 7.0

We are experiencing mapping explosion with number of fields upto 4500 and increasing. Then the first thing which came to our mind was to implement nested fields to reduce the number of fields. With the help of nested fields we could reduce that number to 100 and hardly will increase beyond this.

But nested fields comes with it's own disadvantages like,

1. number of documents will increase
2. Kibana doesn't support it. so no simple visualization and cannot search on that field from Kibana (Discover tab)
3. Performance problems because every query on nested fields involve a join.
4. Limit on number of nested objects (10000) across nested fields of an index.

This seems like a lot of side effects of using nested fields to solve our problem.

In search of an ideal solution, we realised that all our documents doesn't have all the fields! So we are tempted to index the similar documents( in terms of similar fields) in a specific index and rolling over to new index when the limit is hit. And each time we index a document we could check what's the best destination (index) for it. (In terms of similar fields).

But the thing is some of our documents alone has more that 1000 fields. So in this scenario we would hit the limit no matter in which index they end up in.

So what could be the ideal solution without many major side effects like inability to visualize/search from Kibana?

And also if we implement the above solution we thought of, does searching on an index pattern fail? As even though the number of fields of an index will be under 1000 but the index pattern will span on many indices thus resulting in many fields per an index pattern. Is this a issue we are supposed to be concerned about?

Thanks,

[spinscale (Alexander Reelsen)]

Hey,

so the index approach is a first good step to also make sure documents with a stable number of fields stay unaffected from documents with an increasing number of fields.

That said, you might want to take a look at the new flattened datatype, which has just been added in Elasticsearch 7.3.

--Alex

[elk11]

Thanks for the response @spinscale

I see it's a xpack feature. We currently use OSS version of elasticsearch.

Is this this datatype available in basic license (which is free) too? And since we are on the subject, does all of the XPACK features available in basic license? If only few of XPACK features are available, is it documented somewhere?

And is this datatype supported in Kibana visualisations?

[spinscale (Alexander Reelsen)]

yes, this is part of basic.

[elk11]

@spinscale flattened object is great way to solve this problem with the limitation still being not able to visualize in Kibana.

But is there solution similar/alternative to this in OSS version of elasticsearch? Because we would love to continue using Open source version of elasticsearch.