Cloudflare Logpush integration

Elastic Stack Elastic Agent
1

Hi,

we're ingesting data from MinIO bucket using Cloudflare logpush integrations, due to the huge amount of data sometimes the data flow is not "real time" because a single agent can handle the data, using more than one agent seems like not collecting data at the same pace.

Is there any way to parallelize or optimize data ingestion?

Thanks,
Alessio

2

Are you using the Cloudflare Logpush integration with the S3 polling mode, and configured it to poll your bucket on MinIO, right?

The main issue with the polling mode is that it does not scale and you cannot parallelize it.

Adding another agent would probablly lead to duplicate data.

To have multiple agents consuming this data it is required to use SQS, but I don't think that this is possible since you are using MinIO and not AWS.

Is that just one site or you have multiple sites sending logs to the same bucket?

If you have multiple sites sending log to the same bucket you could create one bucket per site, and then use one integration per bucket.

3

Sorry for the late answer! Thank you very much!