I have a React/Django site focused on a search application built with Search UI. Right now I have it set up to query an App Search engine with a small test subset (~1000 documents) of the full dataset (~550,000 documents and growing daily).
We're beta testing our search app, so we're changing the data and interface pretty frequently. I've found working with this many documents in App Search to be pretty unwieldy (ie, the difficulty to delete documents other than by ID or by removing an entire engine).
I've also had trouble installing the Logstash plugin to ship documents to my App Search instance, apparently due to recent Sinatra updates incompatible with Logstash (discussed in this Github issue).
I've read that it isn't advised to connect Search UI directly to Elasticsearch (mentioned here in the docs and here in another discussion post).
I'm curious what exactly is cautioned against here - I'd like access my entire Elasticsearch index via my search application, so I'm not worried about only giving end users access to some of the data in my instance. I'm also authenticating users on my site, so I'm not worried about unlimited public access to my dataset.
Any advice for managing a large dataset in App Search or guidance to safely configure Search UI to query Elasticsearch would be a huge help, thanks!