Hello! I'm currently developing an autocomplete feature for my mobile marketplace, based on my products dataset, stored in my main database.
I currently have about 100K documents loaded on my products index using logstash. The products are managed by houndreds of vendors, so everyday I have a lot of additions and removals in my database that I need to sync with app search. Updating the index on a daily basis seems fine for me.
I made some research regarding how to keep indexes updated on ES, and found two different approaches. Since a massive deletion with logstash seems to be resource intensive, one approach is to create a new index every day and dump the older one, and another is to use the new index lifecycle management.
That may be fine when working directly with ES, but I found that AppSearch creates many indexes with custom names. I'm pretty sure that if I try to manipulate those indexes directly in ES I will break AppSearch.
On the other hand, I didn't found anything about index management in AppSearch.
What strategy do you recommend to keep the indexes updated on a daily basis?
Just in case, I'm running the whole ELK in my own docker environment.
Thank you very much!