I'm building a search engine for my app. For that i'm trying Elasticsearch for the first time. My app uses a Postgre SQL. As I understood I need to replicate datas from my Postgre to Elastic Search in order to perform REST searches on it.
I understood that I need to configure a replication stream whenever my Postgre gets updated thanks to Postgre's locical decoding feature. I found few tutorials that recommend using Debezium for data capture and Kafka for transport. But i'm a bit lost on what to build and how to build it.
All my infrastructure runs on Google Cloud. I enabled the logical_decoding flag on Cloud SQL. I created a ready-to-use Elastic Search and Kafka instances thanks to Google Cloud partnership.
But now I'm kind of lost on how to bundle everything. Should I run an additional Debezium server to capture Postgres changes ? The configurations I've seen seemed a bit overkill for what I'm trying to achieved. Is there any built-in solution on Google Cloud (Is Cloud Fusion suited for this and for production ?) ? I've seen a Postgre plugin on Kafka that seems to be able to capture Postgre data, Is that a solution ? How do I securely connect everything together ?
I'm not even sure I'm going in the right direction... Is there any kind soul around here that could take me by the hand and describes the best stack to achieve a search engine from a Postgre database on Cloud SQL (No web scrapping possible). I'm a fair developper but a learning DevOps.
Thank you very much !