Last Valid Value of a Field in Scripted Fields

Elastic Stack Elasticsearch
1

Hi

I need to perform math functions on fields with different timestamps. For example, current value of field x, qualified by its size and the last valid value of field y:

if(doc['x'].size()>0){

 doc['x'].value+doc['y'].LAST_VALID_VALUE; 

}

Note that x and y have different timestamps and doc['y'].size() will not be > 0, when doc['x'].size() is > 0.
Any thoughts?
Cheers,

2

Are x and y part of the same document or are they stored as individual docs? You say, they have different timestamp, which makes me think you talk about individual docs.

I think it will be easier for me and others if you can provide an example with input doc(s) and the expected output.

3

Thank you for your reply. They are part of the same index; however, different document, i.e., were received through 2 different jsons, e.g., {"x":vlaue_1, ...} and {"y":value_2,...}.

4

In this case you won't be able to solve your use case with a scripted field, a scripted field can not work over 2 documents, but only within the same document. With other words, a scripted field would only work if x and y are part of the same document.

In order to combine docs you need aggregations. Do the docs you want to combine share a field with the same value, e.g. an id? In this case you would group the docs using that id (e.g. with a terms aggregation) and use another aggregation to combine x and y.

For large data sets, you need a composite aggregation, if you aim for persisting the result in an index again - as a new document - you can use a transform.

5

Thank you for your prompt reply. Documents do share a common unique field, would you be able to point to an example, for aggragation templating purposes?
Cheers,

6

You can check this discuss post, this is about merging 2 documents. Conceptually you want the same but in addition you want to apply a calculation. In a nutshell you need to get the values for x and y in the map phase and do the calculation in the reduce phase.

If you haven't worked with aggregations before: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html

If you want to create new documents, checkout transform: https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html

I know this is a lot to read. It might be easier to look for blogs first, e.g. https://www.elastic.co/blog/intro-to-aggregations/ or this webinar recording about transform: https://www.elastic.co/webinars/introducing-data-frame-transforms-for-elastic-machine-learning

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.