Is it possible to buffer / cache / store scripted fields in ES?

Elastic Stack Kibana
1

Hey there,

I would like to ask for your support as I am currently failing to find a solution for storing scripted fields in Elastic Stack.
The setup is plain Elastic Search and Kibana (without Logstash).

My use case is as follows:

  • I have got a few million datasets of several fields, of which one field contains article names as a String
  • Using scripted fields I check the article name and if it contains a certain prefix, I will split it into its components (attribute 1, attribute 2, ...)
  • Those attributes will then be used in Visualizations

Unfortunately, Kibana seems to recalculate all the scripted fields with each change in the Visualization.
E.g. changing the filter will take some time to display the new results.

Considering that the dataset does not change, I would assume that the scripted fields' values don't change either and would only need to be calculated once.
However, I am at a loss regarding how to buffer / cache those values and can't find a method to write them back into Elastic Search.

I would like to avoid any changes to my ingest workflow.
Based on Scripted field with stored script I would assume that there is some way to add scripted fields in Elastic Search or write the results of scripted fields back into Elastic Search's index.
So far, my research into how to accomplish this has come up empty regarding.

Thank you very much for any insight :slight_smile:

2

The recommended way of handling this is to do the calculation of the scripted field before ingesting the document into Elasticsearch. You can do this for example by defining an ingest pipeline with a script processor: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/script-processor.html

For existing data you would have to re-ingest the documents through this pipeline.

1 Like
3

Thank you for this hint :slight_smile:, this seems to be what I am looking for.
However, I am unable to get it to work.
My problem seems to be that I have a script consisting of several lines and if-clauses, and am currently unable to use it with the script processor.
Preferably, if possible, I would like to copy&paste my painless scripts into the script processor, do minimal adjustments - e.g. switch out the "doc['A.keyword'].value" for "ctx.A" - and have it run this script.
Yet, it's either my lack of proficiency with the script processor or I am using a way of thought that Kibana / ES does not like.
Could you please have a look?

One of my current scripted fields does look at two fields (A and B) and checks whether they conform to a predefined nomenclature.
If both checks pass, the scripted field should output "true" - if not, then "false".

My current painless code for the scripted field is as follows:

    if (doc['A.keyword'].size() == 0 || doc['B.keyword'].size() == 0) return false;
    def indicatorName = doc['A.keyword'].value; 
    def indicatorB = doc['B.keyword'].value; 
    if (indicatorName.startsWith("ABC") && indicatorB == "A1") { return true; } 
    return false;

Thank you very much in advance.

4

How does your request putting this into the pipeline definition look like and what's the error message (if there is one)?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.