Integrate Elasticsearch with PostGres in nodejs app

Elastic Stack Elasticsearch
1

Hello!

I'm kinda new to Elastic Search(I'm a noob in this particular technology) and I would like learn more about Elastic from the community. I'm building a nodejs app using PostGreSQL as the database and I want Elastic Search as a Search Engine to this Node.js app.

Could someone guide me through the process? Although, I've been a web developer for a really long time, I've never thought of using Elastic so far, now I need this search features for my nodeJs app.

I've already installed Elastic on my Ubuntu machine, I'm finding it hard how to integrate both Elastic and pgsql. A blog post or a reply to this query would help me a lot.

Thanks a bunch for taking the time for reading my question

2

You need to send your existing data to elasticsearch.
That means:

  • Read from the database (SELECT * from TABLE)
  • Convert each record to a JSON Document
  • Send the json document to elasticsearch, preferably using the _bulk API.

Logstash can help for that. There's also a PostgreSQL connector that might help.

But I'd recommend modifying the application layer if possible and send data to elasticsearch in the same "transaction" as you are sending your data to the database.

I shared most of my thoughts there: https://david.pilato.fr/blog/2015-05-09-advanced-search-for-your-legacy-application/

Have also a look at this "live coding" recording.

2 Likes
3

also relatively new to unstructured DBs - check out ZomboDB and see if it meets your needs.

I was very stoked when my AI research assistant came up with that moments ago! As we're pushing our first prod API using elasticsearch this week --yay vector db magic-- this find was well timed. Works with Kibana, too.

We're not committed yet, there's at least 3 primary ways to do this (Logstash, pg_sync, ZomboDB) but we'll get there soon. Still looking for clarity as to expected performance and costs at scale for our use case (lots of rows, lots of columns/features, handful of columns max with long text to be searchable).

Good luck, we've got this and hi Elastic community!

2 Likes
4

Nevermind ZomboDB. Will probably just sync normally --I want that full ELK vision! Never hesitate to pay for charts if they're cool, smart charts. (pennies worth of volume hehe)

It's between Azure and AWS for us now. Supabase, Neon, Xata all seemed real cool (in that order based on our 'wikked speed for cheap' priorities) but we had a breakthrough with a gather/generate datasource and pricing over 10gb storage looks spooky with those i.e. we'll just solve for cache on day 1. More power to all 3 (and Timescale) bringing pg-plus-:fire::fast_forward: to the generative generation.

"NOW GET IN MY DB!!!"

5

Hey Hi!

Your words are very inspiring, what are you guys working on? I would love know what you're building, may be I could also contribute to your project. Care to talk about it?

(please feel free to DM me, here in this forum)