Limit the Transform scope based on timestamp

Elastic Stack Elasticsearch
1

Hi!

Supposedly I have logs, with timestamp for each one of them.

Those logs can be group by a few dozens and describe a "session", which can take between 1 second - 1 minute.

I currently have a Transform who is grouping those logs (thanks to a pivot and multiple group by).

Throughout the day, I'm adding logs in Elastic, and they are treated by the Transform.

But, the Transform will always use the old data for the grouping that I don't need anymore.
Because a sessions is like 1 minute maximum, so the events of few hours ago are useless.

I saw that we can change the scope of the source query according to the documentation right here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-scale.html#limit-source-query

Is it something like this?

POST _transform/_preview
{
  "source": {
    "index": "test_v*",
    "query": {
      "bool": {
        "must": [
          {
            "range": {
              "@timestamp": {
                "gte": "now-1d/d",
                "lte": "now"
              }
            }
          }
        ]
      }
    }
  },
  "dest": {
    "index": "my_transform_test"
  },
...

For this example I used the ingestion @timestamp, for the last 24 hours.

Is there a way to use the field "timestamp" (without the @, it represents the timestamp inside the log) in order to change the scope of my Transform?

I prefer the log timestamp because for a given timestamp, if I do minus 1 hour, I'm sure to get the whole session.
But for the ingestion timestamp, if I didn't added log the last hour, then the grouping will fail.

The problem is, I don't know how to change the "now" for "most recent timestamp you currently have".
Like this, I will be able to deal only with the latest data.

I might not be very clear, tell me if you need additional information, thanks for your time!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.