Elasticsearch as producer/consumer solution

Elastic Stack Elasticsearch
1

Hello Community!

Is it possible to use Elasticserach in a producer / consumer scenario?
The producer writes documents to Elasticserach.
The consumer reads documents from Eleasticserach and processes them.
The order of the documents must be preserved.

There is a REST API GET operation but the document's _id must be specified. But the consumer doesn't know the _id of the document, he wants to just get the next document that came.
I did not find an API allowing to subscribe for incoming documents.

Any suggestions?

2

That sounds like a bad fit. Why are you considering Elasticsearch instead of a message queue?

3

Because Elasticsearch is also a database that would keep original versions of documents for later use. In contrast, queuing systems (such as RabbitMQ) are for communication only and do not have a database.
I guess that's a bad idea - but I wonder if it is technically possible to use Elasticserach instead of proper message queueing system.

4

Use Kafka then. It allows you to define retention period. I would not try to build messaging system on Elasticsearch.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.