Incremental Sync for S3 Connectors

Elastic Search
1

Hello Elastic,

I want to confirm on content sync for S3, can we do incremental sync in S3?

Or does S3 only applicable for full document sync?

I do notice that for incremental sync not all connectors support, and it seems like for sharepoint only available on the incremental sync.

I need confirmation on this asap, if S3 does not support on the incremental sync may I know when will it available for S3?

Thanks!

2

Hi @aisyaharifin

You're correct; S3 connector doesn't support incremental syncs right now. We only support incremental syncs for the SharePoint Online connector.

if S3 does not support on the incremental sync may I know when will it available for S3?

I don't think this is on our roadmap but I'm confirming with the team just in case. I'll get back to you when I have an answer.

3

Hi @aisyaharifin, incremental sync support for more connectors is on our roadmap, but S3 did not make it on the immediate roadmap. If you're a customer, filing a gap would help with with prioritization.
Thanks!

4

Following up on this, have you looked at our Extraction Service? The service can be used for extracting contents of large files on your own infrastructure when using self-managed connector clients.

Instead of sending the file as an attachment to Elasticsearch, the file’s content is extracted at the edge by the extraction service before ingestion. The extracted text is then included as the body field of a document when it is ingested.

Maybe this would be helpful for your use case?

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.