Unable to load more than 30000 fields into index pattern

Elastic Stack Kibana
1

Hi everybody!

I have index pattern let's say
myindex_* which includes many other indexes like:

myindex_client1_*
myindex_client2_*
myindex_client3_*
myindex_client4_*
etc...

the indexes like myindex_clientN_* have about 200 to 500 fields. Some are the same for every index some are unique for the specific client. This all add up to more than 32 000 fields in the myindex_* index pattern.

Right now the myindex_* index pattern shows just about 4800 fields. When I try to refresh the index pattern it did looks like it did nothing still showing just about 4800 fields, but when I click on the let's say Scripted fields the page got refreshed and the Fields tab suddenly shows more than 32 000 fields.

But when I again click to Discover or Dashboards and than go back to the Index Patterns and click on myindex_* there is again just about 4800. It never gets saved with the 32000 fields. I was getting 413 response code in the Kibana logs, but I solved it by adding http.max_content_length: 500mb to elastic config on every node and restarting the whole cluster. Now there is no errors in the kibana or elastic logs but it still behaves the same.

I am using this docker image for Kibana
docker.elastic.co/kibana/kibana-oss:7.9.3
and the elasticsearch is also 7.9.3 version.

Can you please help me to save those fields?

Regards
Tom

2

Welcome to our community! :smiley:

I can't help solve your immediate issue sorry.
But I would suggest that your field count is excessive and likely to cause you other issues in the future. You should really address that.

3

I agree- this is an excessive number of fields, and it will slow down Kibana if you try to do this. I'm testing locally with 4000 fields and I keep finding new performance issues.

If you really need to show this many fields, I have some suggestions:

  1. You could upgrade to 7.11 or 7.12, where we automatically pull the field list from the mappings when needed. This will prevent the issue where you can't save.
  2. You are probably running into the kibana server.maxPayloadBytes limit: Configure Kibana | Kibana Guide [7.12] | Elastic
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.