Queue with elasticsearch

[Ahmed_Benneji1 (ahmed benneji)]

I want to implement sort of a queue with the data on elasticsearch, to put you on perspective, the user will launch a query to elastic, and then retrieve, let's say top X results, once those are retrieved, the next user, when they execute the same query, they should get the next top X results, so on and so forth, making query results searchable only once.
So I'm wondering if it's feasible with elasticsearch, (with out over engineering it) in the first place, and if so, any help with implementing it ?

[gabriel_tessier]

Hi,

One simple way to do is to prepare 2 query syntax with all the filter e.g.

query = {"query": {"bool":{"must":[{"some_filter": ...}....]}...must_not : {"term": {"exclude": "False"} }}}

Call1: you run the query for the the search.
Call2: Then you run the same again inside a update by query where you'll add exclude to True.

This way the Call1 will return your result
The Call2 will run exactly the same query with the same result but to update them and add the exclude flag to true, so the next time you make Call1 the result for the first user will not hit.

It's simple enough I think as it add only one call to update by query and one filter in your search query.

Hope it help

Note: there's limitation as it not handle the case if 2 people query in nearly same time, i.e user2 fire a search before that user1 update query run.

[DavidTurner]

@gabriel_tessier

Note: there's limitation as it not handle the case if 2 people query in nearly same time, i.e user2 fire a search before that user1 update query run.

I think I'd call this a bug and not merely a "limitation". Fortunately there is a solution.

I can't promise it'll be possible to achieve amazing throughput, or give a precise FIFO guarantee, but the basic idea of a queue should work ok using this optimistic concurrency control mechanism.

[gabriel_tessier]

@DavidTurner
I'm aware about version but

@Ahmed_Benneji1

(with out over engineering it) in the first place,

I tried to provide a simple solution and notice about what I called a limitation (If I call it bug he'll run away ) so the user can come back and ask again, if he have 1 search a day he's far to be concerned.

Also I thought to use percolate but as I never used percolation I don't know if it can help here.