Cloudflare Integration problem with delayed ingestion

[yago82]

Hi everybody,
using Cloduflare integration I noticed that if the machine where the elastic agent is installed falls for a certain period, Elastic is not able to recover the accumulated ingestion delay.

I'm talking about 1000 logs per hour so a minimum flow.

I tried to change the interval of the PULL, but apparently the interval is both the frequency with which the PULL is performed and the maximum interval of logs taken.

Example:
If i put an interval of 1 hour, it retrieves logs only between 5.00 to 6.00. Then after an hour it retrieve from 6.00 to 7.00, but if i have a delay of two hour it never recovers this delay and considering 1000 logs per hour i don't think it's an ingestion problem.

I'm using Elastic Cloud 8.3.3

Thank you.
Regards.

[yago82]

Hi,
an update, for the moment I solved by restarting the elastic agent every 5 minutes.

Thank you
Regards

[jamie.hynds]

Hi @yago82 - I assume you are using the Cloudflare Logpull integration? If so, we recently shipped a new Cloudflare Logpush integration which is Cloudflare's preferred approach to ship logs. While Logpull is still supported, you can likely get improved performance and broader event coverage from Logpush. I'd recommend going down that road, but if you can't - more than happy to dig into the issues you've faced with Logpull.

[yago82]

Hi @jamie.hynds

thank you for your answer.

As you said, I have to keep using the LogPull. For the moment it hasn't given me any major problems. My concern about delayed ingestion. As you can see, the workaround of turning the elastic agent off and on again is not a very "elegant" solution.

I don't understand why the integration fetches the logs of an interval equal to the frequency of the Pull. If set every 5 minutes, he retrieves logs of an interval of 5 minutes and no more.

Thank you