I have a transform that aggregates my stats index data for long-term storage, but the results are no longer correct. My stats index stores each request (one document = one request) and stores the credits used by that request, e.g. 6. I transform this into one-day aggregations, e.g. customer A did 1…

You specified a query filter that limits transform to only access data from the last 3 hours, actually even just 1.5 hours, because of the 1.5 hour delay: "must": [{ "range" : {"timestamp": {"gte" : "now-3h"}} }] Such a filter is never a good idea. Transform tak…

Discuss the Elastic Stack

Transform missing data

Elastic Stack Elasticsearch

rokcarl (Rok) July 14, 2022, 2:12pm 3

Thank you, that was it and it fixed my problem.

I'm facing a new one though about data mismatch, but I opened a new thread for that.

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.