Calculating fields in logstash on upsert

[tumbledwyer]

Hi

I have a pipeline used to enrich some json data from different sources to build up an index for a report. I would like to do a calculation based on 2 fields from the different sources, but since the sources are different, the fields are not in the same event so I want to use a script to do the calculation on upsert.

My mapping looks like this:

{  
  "properties": {
    "objectA": {
      "properties": {
        "fieldA": { "type": "date" }
      }
    },
    "objectB": {
      "properties": {
        "fieldB": { "type": "date" }
      }
    },
    "daysBetween": { "type": "integer" }
  }
}

objectA and objectB come in as 2 different events and a common field is used to link them to the same report object. This works just fine when using doc_as_upsert and both object A and objectB are in the index. But now I need to calculate the daysBetween field and I've tried to use the script in the elasticsearch output to do the date diff but now my upsert is broken and only the last object that arrived is in the index.

So my logic should look something like this:

- No object in index: event=objectA
    =>  upsert objectA
- objectA in index: event=objectB
    => upsert objectB
    => add field daysBetween=datediff(fieldA-fieldB)

...and vice-versa if objectB arrives first

How would I go about doing this?

[Badger]

If you are asking how to do a scripted update that is really an elasticsearch question, not a logstash question.