Scripted fields performace

Elastic Stack Kibana
1

Hi,

We had lot of fields in our elasticsearch documets due to which we had mapping explosion. To solve this recently we implemented flattened object which helped us to reduce the number of fields drastically.

post this visualisation broke because Kibana doesn't support flattened objects in visualisation.

Since there are about 200 fields which are being visualized, we were thinking of adding scripted fields for each of those 200 fields as a workaround to be able to visualize on them.

So will there be any serious performance issues with adding so many scripted fields? Can i configure somewhere such that scripted fields aren't calculated for 'Discovery' tab searches and only used in visualizations? (To avoid calculating scripted fields for every search)

Is the support for flattened object in visualization is something Kibana considering?

Thanks,

2

We're tracking this at https://github.com/elastic/kibana/issues/1084#issuecomment-532996333, the short answer is it's being worked but not available in the near future.

There will be a performance impact, but there's a lot of factors to consider. Probably the most important consideration is the number of documents operated on in the aggregation and the complexity of the script. It'll be a linear with the number of documents if you're using it to flatten for each query, which is why we usually recommend doing it pre-ingestion. With that said I wouldn't expect it to have a significant impact. It may cause a stressed cluster to tip over, but ideally the cluster isn't stressed. To get a concrete answer I think you'll have to run some performance tests, but I think it's worth a try.

Discover will pull 500 documents at a time so I wouldn't expect it to have much of an impact. We can limit the amount of documents it pulls in in advanced settings too, if that's a concern. You can also click on the "source" tab in the index pattern and filter these out from discover searches too.

3

Thanks for the reply! I tried to add all those fields (200) as scripted fields but got some error about max compilation rate (default 15/1m). I had to increase it to 1000/1m for all the different queries to work (and also increase the timeout value to 90000ms)

Everything is working as of now although it's all slow. But is it a dangerous thing to set the compilation rate so high?

You can also click on the "source" tab in the index pattern and filter these out from discover searches too.

i tried doing this. But turns out you can't add scripted fields to source filtering. (it shows "The source filter doesn't match any known fields.") And the scripted fields still appear on discover page. is there other way to avoid computing scripted fields for all queries other than visualizations?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.