How to subtract two values in elasticsearch

Elastic Stack Elasticsearch
1

if my logs looks like this,
{ name: "A1",X: 10}
{ name: "A1",Y: 8 }
{ name: "A2",X: 15}
{ name: "A2",Y: 1 }

then how can I subtract (X-Y) which have same name?
What I want is,
for name "A1" X-Y = 10-8 = 2
for name "A2" X-Y = 15-1 = 14

2

You can not as it would mean joining documents, which Elasticsearch does not support. Instead try to merge these into a single document.

3

All these logs are in same index

4

Does not matter. It has to be within the same document.

1 Like
5

how to merge it into single document?

6

Do that when you are indexing data. Exact how will depend on your use case.

7

can you share any link for that?
I am using fluentd to send logs to elasticsearch.
I was trying scripted field in kibana, but it didn't work.

8

You will need to combine documents or update the first document with the second. Not sure fluentd can do that.

9

Adarsh,

One way you can merge (with side effects) is by making "name" field as _id and X / Y as fields. You can use script processor to compute (X-Y) at ingest time or use scripted_fields to compute at query time.

Note that as your index grows, your ingestion throughput will suffer badly. Your data may not distribute evenly. If there are other fields in the index, design may affect other queries.

Doing it from outside ES will require tool to deal with optimistic concurrency control (if_seq_no or if_primary_term). Read document with name = A update if_seq_no = X. If document not found use create instead of update. This too will slow down ingestion.

10

Fixing at ingest is still be best solution, however if it's not possible to get both values in 1 document, but you can't avoid having 2, you can use transform to pivot the index and calculate what you need.

With transform you can group_by documents by name and e.g. use a scripted_metric aggregation to get X and Y and calculate X-Y. I do not have a concrete working example at hand, but the examples should be helpful to craft something.

2 Likes
11

Thanks everyone!!
Finally did that by merging logs by names using fluentd concat plugin:


and then send the logs elasticsearch.
1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.