Appsearch support for large attachments

[Michael_Sor]

We are trying to figure the best solution for handling attachments with Appsearch. I saw a thread regarding a hard 10MB limit for file. To overcome Appsearch limitation is to build a custom intermediary solution to parse files and handle either with Tika or use elastic-search attachment ingest and then parse file into JSON and send to Appsearch.

Wanted to know what approach people implement and any issues/constraints we should consider.

Reference:

Ingest meta data with attachment in App search

hi Team In the elastic, we usually ingest the attachment in elasticsearch using attachment plugin(Ingest Attachment plugin | Elasticsearch Plugins and Integrations [master] | Elastic) so we don’t have to worry about the types of file we are trying to ingest. Typically, we need different types of library to read the content of different types of file(i.e work, excel,pdf,pptx). By using this plugin, we do not need any types of library to read the content. Can we use the same with app search?

Index the entire file content

In Appsearch do we have a datatype called "Attachments" similar to Elastic Search. In Elastic search this is handled by installing the File Ingestion Plugin. This uses Apache Tika for Content Extraction. If we have this in Appsearch, this will be helpful to index the entire file content.

[ggbertelli (Gigi)]

I did not see the datatype called "attachments" in the Appsearch documentation

I found a suggestion to use _simulate (which means to post the attachment to elasticsearch first then query the elasticsearch _simulate to get back the data to be submitted to the Appsearch

How to index PDF file data and search data from attachment PDF file

So you need to add the ingest attachment plugin Click on Settings and Plugins: [image] Add the plugin. And don't forget to save the changes. [image] After the cluster has been updated, you will be able to use the Elasticsearch endpoint to call the _simulate API. See Simulate pipeline API | Elasticsearch Reference [7.11] | Elastic If you mix that with the plugin documentation, you should be able to execute something like: # Create the pipeline PUT _ingest/pipeline/attachment { "descript…

or

Is it possible to index Files (PDF, DOC, PPT) using App Search?

I would like to use App Search for searching files from the server folder, Is it possible to feed this to Elastic Enterprise Search - App Search as a document using ingest-attachment-plugin. My server is NodeJS, also I prefer App search as it has options like Synonyms, Curations, ... Help will be appreciated, Thank you

Are we missing something?

Another solution is to use Tika: * programmatically parsing my files with Apache Tika (should I convert them to Base64?)

Thank you in advance

[dadoonet (David Pilato)]

There is a feature request about connecting FSCrawler with AppSearch but it's not yet there.

Allow connection to AppSearch service

[Michael_Sor]

With appsearch limits, is there anything to bypass results per query.

• Is there a way to bypass the 10,000 paginated results limit?
• Is there a way to get all IDs for given Elasticsearch request?
• Why is search limited to 10,000 records if Result Pages limit is 100 pages and Results per page limit is 1000?

[dadoonet (David Pilato)]

Not sure this applies to AppSearch but with Elasticsearch you can use:

• the size and from parameters to display by default up to 10000 records to your users. If you want to change this limit, you can change index.max_result_window setting but be aware of the consequences (ie memory).
• the search after feature to do deep pagination.
• the Scroll API if you want to extract a resultset to be consumed by another tool later. (Not recommended anymore)