Hide/Disable runtime fields from Kibana available fields

Elastic Stack Kibana
1

Hi,
Is there a way to hide/disable/pick certain runtime fields from available fields in Discover? When applying many runtime fields, Discover page takes forever to load due to the number of available fields, While Elasticsearch returns the response very quickly. For example, I took 50000+ runtime fields and applied a search using dev tools which runs as expected:

image
image750×465 14.5 KB

However when going to Discover - despite the response returns quickly it takes forever to load due to Kibana trying to get all available fields:
image
image1922×767 49 KB

Runtime Fields is a great feature that helps to deal with unpredicted users data but it is not well adapted for Kibana since it may cause Discover page to be impractical.

Thanks,
Ofir

2

I believe page loading time is definitely a valid concern. I'll open an issue to follow-up on this. In the meantime, you can exclude certain fields from a data view by specifying field filters. Go to Advanced Settings > Data Views (or Index Patterns if on a version < 8.0.0) > select your desired data view > Field Filters.

3

Thank for the advice. This is quite helpful and loading time is much better when filtering fields in index patterns (and i can see i can still search in Discover based on a field and also view JSON Source). I would appriciate if you can paste a link to the issue in here.

4

What's the order of magnitude of your runtime fields? 10s? 100s?

5

Just realized you had already stated it was 50+k.
Here's the issue.

1 Like
6

It varies. But i saw a client with 100,000+ runtime fields for which Elasticsearch worked fine but he could not use the Discover page in Kibana due to available fields. He did not use most of the fields (mostly the index ones) and documents had just some of the runtime fields (in general there are many but data varies largely between document).

7

I think with the order of magnitude of 100k fields, there's not much we can do. We can't know upfront which ones are being used.

8

The option you provided me with is satisfying for me :slight_smile: I wanted to know if there is more holistic approach for example: "hide all runtime fields" either in index-pattern or on in advanced settings.

9

Unfortunately not as of now.
If you switch to readFieldsFromSource in Advanced Settings > Discover, you won't be able to access runtime fields at all. But this switches to reading from _source directly, which is something we'd like to get rid of.

10

In my case the runtime fields are available on _source but i'm using dynamic: "runtime".

11

Sorry, but those two are not the same thing. By setting the dynamic to true (or runtime), you are telling Elasticsearch to add fields to an index based on their presence in the documents, although they are not defined in the mapping. They WILL be indexed.
Runtime fields are fields available on read. They can be specified in the mapping as well, but they WON'T be indexed.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

13

This took a while, but I'm pretty sure this is the issue and we're preparing a fix for it [Discover] Bad performance rendering document table when mapping contains lots of fields · Issue #144673 · elastic/kibana · GitHub