How rollup jobs memorize last processed document?

Elastic Stack Kibana
1

I went through my rollup job JSON file and I think current_position in below config is the way how a rollup job memorizes the last document it processed.

"status": {
    "job_state": "started",
    "current_position": {
      "@timestamp.date_histogram": 1571728849000,
      "status.terms": "complete"
    }

My question is what if I insert data which have @timestamp value less than @timestamp.date_histogram (last processed timestamp value)

Will they also be processed?

Or @timestamp values which are greater than last processed timestamp value will only be processed?

1 Like
2

@alisongoryachev can you please help here?

Thanks,
Bhavya

3

Hi @isharamadhawa! The timestamp is important. It's used as the after_key. If you add, change or delete data before that timestamp it won't be rolled up again.

To compensate for ingest delay, you can use the delay parameter. See the Create rollup job example and the Rollup job configuration documentation for more details.

2 Likes
4

Hi @alisongoryachev, Thank you for your answer :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.