Continuous tranforms stop indexing after initial indexing

Elastic Stack Elasticsearch
1

Hy guys,

I try to setup a continuous transforms strategy on a new cluster for optimize queries and index size with an 1h aggregation.

The stack is setup like that :

  • 1 index input-[0_9]{6} with a write alias "input". A lifestyle policy is attached to split the index when it size reach 100Gb. Logstash put datas to the alias index "input". Refresh interval of index is set to 30s.

  • 1 index aggr-[0_9]{6} with a write alias "aggr". A lifestyle policy is attached to split the index when it size reach 100Gb. Refresh interval of index is set to 10s.

I setup 10 transforms for various datas with source "input-*" and destination "aggr" and I have setup a pivot group by date_histogram on field @timestamp with 1h and the refresh sync is linked to @timestamp with a delay setup to 60s.

So after create all of my transforms I start all of them. At this point I identify 2 phases on transforms :

  • indexing state : transforms indexes all existing datas on indexes "input-*"
  • started state : transforms is entered on continuous mode and check new data

So on indexing state phase it have no problem, all transforms works and I can see datas on indexes "aggr-*". But when transforms passed on "started" state no more datas are added... No error, and transforms just stay on "Finished indexing for transform checkpoint [0]." of the previous state.

I'am on elasticsearch 7.8.0. It's a bug ? What's can be wrong on my setup ?

2

Ok I think I understand, the checkpoint use the current date to compare data ? If this is that, this make sens that it's not working when dates on document are older...

3

That's correct. Continuous mode requires a timestamp that is real time-ish. The source must be append-only, updates of existing documents is not supported.

4

Yeah, fixed by adding "current_date" field on records from logstash

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.