Ingest pipeline analyze time difference

Elastic Stack Elasticsearch
1

Hello,

I need to route messages which have @timestamp field that differs more than 8 hours from now to a separate topic.
I tried to use ingest pipeline with some processors for that. But the only one which comes to my mind is a script processor.
Have anyone used a script processor for such modification/routing?

2

I assume that by "topic" you mean "index"?

I would use a combination of two set processors. One that sets the value of a field to the current time, and one that conditionally sets the index if the time difference between the two timestamps is more than 8 hours:

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "ingest_time",
          "value": "{{_ingest.timestamp}}"
        }
      },
      {
        "set": {
          "field": "_index",
          "value": "some_other_index",
          "if": "ChronoUnit.HOURS.between(ZonedDateTime.parse(ctx['@timestamp']), ZonedDateTime.parse(ctx['ingest_time'])) > 8"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "@timestamp": "2019-11-07T20:39:00.000Z"
      }
    }
  ]
}
3

yeah, I meant index.
Thanks a lot @abdon that is exactly what I looked for.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.