Postgresql to Elastic Search

Elastic Stack Elasticsearch
1

Hi everyone, I'm new to Elasticsearch.

Currently I'm trying to replicate my db from postgresql and transform it to Elasticsearch. I've tried using google datastream (since my db is deployed in cloudSql) to cloud storage, then using dataflow to ingest the output from datastream to Elasticsearch but I need to somehow transform the data from datastream output to csv or pubsub message since the template is only supported in csv / ingest from pubsub. I think it's possible by using cloud function but I feel like there will be too much process. I'm also looking at some psql extension like zombodb, but it's not fit with my need where I want to transform the data first before storing it to Elasticsearch.

Is there any suggestion or recommendation that I can look for ?

Thank you in advance :pray:

2

Hi,
i have done something similar to what you need and maybe i can help you a little bit.
You should try to ingest data into elastic from Logstash, you can use the Jdbc input plugin or the file plugin if you have CSV files.

Jdbc plugin - Jdbc input plugin | Logstash Reference [8.9] | Elastic
File plugin - File input plugin | Logstash Reference [8.9] | Elastic

If you have more question just ask and i can help you a bit.

hope i was helpful

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.