Backfill old data for integration in elastic-agent integration

[liquidkite]

Hello all,
I've been using elastic agent to fetch logs from a variety of data sources. We ran into an issue where there was a misconfiguration with the pipeline and that caused to drop logs for that timeframe. Once the issue wa fixed the logs are flowing but we are still missing data.

Is there a way to fetch the logs for the missing timeframe?

[leandrojmp (Leandro Pereira)]

@liquidkite

Is there a way to fetch the logs for the missing timeframe?

I would say that it really depends on the integration and the input of each integration, but from my experience with Elastic Agent I would say that this is not something straight forward to do, or even possible in some cases.

For example, integrations that uses the httpjson input to get data from API endpoints will have a cursor with the latest data timestamp and you can't easily reset this cursor, basically you have to remove and add the integration again.

Another example are integrations that use TCP/UDP ports to receive data, like some network devices integrations, logs that are dropped will be lost.