May I encourage you reading http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/
Well. I like sending data to both systems at the same time:
- In Postgresql
- In Elasticsearch
Now the question is about "ASAP". What does it exactly mean? How much are you ok to pay for real "real time" instead of the 1s delay by default or more like 2 or 3 s if you are using a BulkProcessor strategy?
It's always a question of tradeoffs.
If you want to have real real time access, then you can slow down elasticsearch and ask it to answer an index operation only after the refresh is done. This has been introduced recently with the new
It depends again on what you have. If you want to index 1 million documents per second, I don't believe that
wait_for_refresh will help. If you have 1 or 2 docs per seconds then I believe it's totally doable.
But note that elasticsearch won't give you back a response until a refresh happened. Which can slow down your "transaction" basically and leave the end user that it's taking now more time then before to save an object.
It depends also on the number of tables you have to update in postgresql for one single object insertion. If it takes already sometime, may be you won't ever notice that you potentially waited for 1 s in elasticsearch?
I'd give elasticsearch a try and see how it works on your real use case. You might be surprised that it's not behaving as bad as you seem to think. (Or I misread your post).