Managing Large Document Uploads: Performance and Limitations in Elasticsearch vs. Standard App Search Engines

[Chenko]

Hello,

We're encountering a challenge with a client who needs to upload large documents using App Search — larger than 10MB per individual document and up to 100MB in total for bulk requests. Our current setup utilizes a standard app search engine, but we're hitting its limit of 10MB per request. On the other hand, Elasticsearch allows for 100MB per request, and this limit can potentially be increased.

I have two key questions:

1. Why does this size limitation exist in standard app search engines? Is it primarily for performance optimization, to conserve storage space, or are there other reasons?

2. We are considering a solution where we use an Elasticsearch-based engine with app search mappings to handle 10+MB documents. Would this approach negatively impact performance, especially in the context of large-scale document handling?

Your insights on these matters would be greatly appreciated, as they will guide our strategy in addressing our client's needs.

[Kathleen_DeRusso]

Hi there @Chenko this is due to performance reasons.

We do have a setting app_search.engine.document_size.limit but this is still constrained by the other limits (no more than 100 documents/no more than 10mb for a single indexing request).

You could try the Elasticsearch-based engine approach with custom settings but you will likely run into suboptimal performance with that route. If you can decrease the size of the documents that need to be indexed, that would be better for overall search performance.