Create dynamic column and save output in ES

Elastic Stack Logstash
1

Here is my question. I have following column which has Int type.

  1. DurationA
  2. DurationB
  3. Duration C = DurationB-DurationA

DurationA and DurationB values comes from Different CSV file which I am loading to ES through Logstash

Question

  1. is there anyway I can create column DurationC and save value in ES while I am processing CSV file ?
  2. if not possible what function I should use in ES Query to get the desired result (DurationB-DurationA)

Thanks

2

can anyone help me on this issue ?

3
  1. You might be able to use an aggregate filter, but I suspect it'll be fragile. Logstash isn't meant for the kind of join scenario that you describe.
  2. ES doesn't do joins either, so unless you have DurationA and DurationB in the same document you're out of luck.

I suspect you'll need some custom glue for this. Logstash can help you read the CSV files but you either need a custom plugin or a custom something else for the join operation. I can't get more specific without more details.

4

Thanks magnusbaeck for your reply. DurationA and DurationB values is there in same document. Once documents loaded in ES , can I update my ES document to populate DurationC ? if yes what aggregate function I can use it ?