Elasticsearch 6.6.1 - Rollup Job stopped adding documents

[Fabio-sama (Fabio)]

Hi there!

I'm experiencing some problems with the rollup jobs.
I'm using it to hourly aggregate data from several indices (abc_*).

It worked perfectly until yesterday at 2pm. Since then, it starts the indexing procedure, the trigger count increases, the 'current position' inside the JSON increases, but it doesn't add any document to the index it used until yesterday (hourly_abc).

Now I created a new job with exactly the same characteristics and it doesn't work either.

Though, data keeps flowing into abc_* indices (in fact the 'current position' is updated properly).

The only thing I changed yesterday is the Index Lifecycle Policy for the abc_* index, deleting them after 24h rather than 6h as it used to be. Though, why should it influence the rollup job?

What can be happening?
Thank you!

[polyfractal (Zachary Tong)]

Hi @Fabio-sama, sorry for the delay, just saw this. A few questions:

• Do you see anything in the server logs about rollups? Warnings about failed searches or something? If there's nothing sensitive in the logs and you are willing to share, I'd be happy to take a look
• When you created the new job, did it stop at the same place? Or just a random position?
• Is there an ILM policy for the rollup index (not the source, but the destination)?

It's interesting that the current position is continuing to update but no new documents. I'm not sure what's going on, but if we need to we can crank up the debugging on the job to DEBUG or TRACE and hopefully get some more information.