Choosing the right path for elasticsearch, logstash

Elastic Stack Elasticsearch
1

I'm not understanding the inegration use case for logstash, weather do I should integrate it or not. I want to check some analytics on my transactions data, API requests, platform, as many as things possible to get the data.

I've already build my application using nodejs & mySql, now I want to elasticsearch & kibana for analytics purpose. So I want to ask how should I integrate,

Weather I should make REST API call to put my data into elastic search?
Or Weather I should sync mySQL Db & elasticsearch with some script.

For which part I should use only elasticsearch REST API, and for which part I should integrate logstash?

Can someone guide me or link me to a good article for the use cases.

2

It depends if you want to have some real-time query or not.

Ie. Do you want to have access almost immediately to the data that the user stored in MySQL?

If so, I'd say that using an ETL like logstash is not the good tool for that.
I shared my thoughts in this post: http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/

But if you are ok with let say 5 minutes latency, then Logstash is good IMO.
Just create a logstash pipeline with an input jdbc plugin and an output elasticsearch plugin which runs every 5 minutes (jdbc plugin parameters) and you should be OK.

If you have specific questions about building that please ask in #logstash channel which is better for that.

3

Thank you David Pilato,
Yes, I do not need to have real-time query, I also do not want my application to slow while updating to elasticsearch for each bit.

Probably input jdbc plugin should work for me.

4

I also do not want my application to slow while updating to elasticsearch for each bit

I don't think that will be the case IMO.
Specifically if you index asynchronously in elasticsearch and don't block any thread.
You can also write to something like Kafka then read Kafka with logstash.

Reading the database every 5 minutes will put some pressure on your database. Also if you model is complex (like split on multiple tables), it might be hard to write a single query to fetch your object.

On the other hand, when your application holds the full object in memory, it's efficient, immediate to serialize in JSON and send the object to elasticsearch.

My 2 cents.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.