Using ElasticSearch as a Microservice

[Mohan_Radhakrishnan]

Hello,
This is a design question. We have messages pushed to Kafka and Kafka can pipe it to Elastic using Kafka connect. This is what I am considering for message search. Kafka can send these messages but it also forwards it to the Elasticsearch service. Messages are multi-lingual.

We also need API access to these messages. This is not search. A screen for a particular user will have messages for that user. Traditionally a message micro service is used for this purpose. But I though I could unify the search service and this API.

Can this be done using Elasticsearch ? I can avoid a separate Micro service for this as all the messages are in Elastic now. I plan to explore how Elasticsearch can be hosted in Kubernetes as all our services are there.

The only reason for have a Spring Data Elastic Micro Service is to check some business rules. Has the message been viewed already ? How to expire the messages ? Has the user been blacklisted ?

Thanks.

[warkolm (Mark Walkom)]

The retrieval of messages for a user would be akin to a search, just with a filter for that user right? So Elasticsearch can do this.

[Mohan_Radhakrishnan]

Yes. But the layer before Elasticsearch has to take care of checking blacklisted customers and other security restrictions. I assume the Elasticsearch cluster in Kubernetes is better if accessed directly instead of a Spring Data Elastic Micro Service. Any architectural pattern guidance is documented in books etc. ?

[warkolm (Mark Walkom)]

Elasticsearch has inbuilt, free security features. It doesn't do access blacklisting as such, but you could implement a list of those in an index that you query first with your code, then anything that passes can run another query to get the info it needs.