Hi!
I have this usecase:
I receive ca 500 documents of cars (brand, model, color, airbag yes/no, etc.) per second and I am storing these in ElasticSearch. It works perfectly.
Then, I have a webclient, which searches for, example, all yellow BMW or blue Mercedes cars with airbags. This also works perfectly and in milliseconds.
But to receive the new cars, the user has to hit refresh. I want the results to popup immediately, so streaming the result to the users browser (via websockets).
Therefore my server logic (java at the moment, but I can switch to whatever I want) would have to poll ElasticSearch every second or so. This does not seem like a good solution to me.
Also, I do not have only one client but maybe thousands, so I am concerned about scaling.
So is there any way of streaming the results to the end user by taking into account, that new documents come in all the time.
If this is not possible with ElasticSearch, do you suggest something else? Apache Kafka for example is excellent at streaming, but (as far as I know), not with filtering the results.
PS / Disclaimer: I also asked this question at StackOverflow ( https://stackoverflow.com/questions/65809522/elasticsearch-stream-new-incoming-documents-to-clients ) , but then I thought I better ask here, where the ElasticSearch experts are.
Thank you!