ElasticSearch Stream NEW INCOMING documents to clients

Elastic Stack Elasticsearch
1

Hi!

I have this usecase:

I receive ca 500 documents of cars (brand, model, color, airbag yes/no, etc.) per second and I am storing these in ElasticSearch. It works perfectly.

Then, I have a webclient, which searches for, example, all yellow BMW or blue Mercedes cars with airbags. This also works perfectly and in milliseconds.

But to receive the new cars, the user has to hit refresh. I want the results to popup immediately, so streaming the result to the users browser (via websockets).

Therefore my server logic (java at the moment, but I can switch to whatever I want) would have to poll ElasticSearch every second or so. This does not seem like a good solution to me.

Also, I do not have only one client but maybe thousands, so I am concerned about scaling.

So is there any way of streaming the results to the end user by taking into account, that new documents come in all the time.

If this is not possible with ElasticSearch, do you suggest something else? Apache Kafka for example is excellent at streaming, but (as far as I know), not with filtering the results.

PS / Disclaimer: I also asked this question at StackOverflow ( https://stackoverflow.com/questions/65809522/elasticsearch-stream-new-incoming-documents-to-clients ) , but then I thought I better ask here, where the ElasticSearch experts are.

Thank you!

2

It sounds like you want https://github.com/elastic/elasticsearch/issues/1242.

3

Hi David,

yes, you are correct, it seems like the linked feature request.
But the request is from 2011 with many many "+1" comments, but it seems like it was never implemented.
So rephrasing it, I would translate it with "We know what you need, but ES cannot offer it at the moment".
This is not really what I was hoping for, but at least now I have a definitive answer and I have to look for something else.

So thank you for your reply!
Bernhard

4

That's right. It's something we'd very much like to implement, which is why it's still open after all these years. We are working towards it, but it's a pretty enormous piece of work.

5

Yes, that I can imagine!

We are working towards it

Do you have a timeframe? I can easily wait 2-3 months, but not 2-3 years.

Thank you!

6

No we have no definite timeframe for implementing this feature. It's certainly more than a few months work.

1 Like
7

Okay, thank you! Good to know!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.