# Choosing the right path for elasticsearch, logstash

**URL:** <https://discuss.elastic.co/t/choosing-the-right-path-for-elasticsearch-logstash/124592>\
**Category:** Elasticsearch\
**Created:** [March 19, 2018, 3:17pm UTC](https://discuss.elastic.co/t/choosing-the-right-path-for-elasticsearch-logstash/124592 "2018-03-19T15:17:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ankibalyan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankibalyan/32/28951_2.png) [@ankibalyan](https://discuss.elastic.co/u/ankibalyan)\
**Post date:** [March 19, 2018, 3:17pm UTC](https://discuss.elastic.co/t/choosing-the-right-path-for-elasticsearch-logstash/124592/1 "2018-03-19T15:17:25Z")

</div>

I'm not understanding the inegration use case for logstash, weather do I should integrate it or not. I want to check some analytics on my transactions data, API requests, platform, as many as things possible to get the data.

I've already build my application using `nodejs` & `mySql`, now I want to elasticsearch & kibana for analytics purpose. So I want to ask how should I integrate,

Weather I should make REST API call to put my data into elastic search?  
Or Weather I should sync mySQL Db & elasticsearch with some script.

For which part I should use only elasticsearch REST API, and for which part I should integrate logstash?

Can someone guide me or link me to a good article for the use cases.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 19, 2018, 3:35pm UTC](https://discuss.elastic.co/t/choosing-the-right-path-for-elasticsearch-logstash/124592/2 "2018-03-19T15:35:39Z")

</div>

It depends if you want to have some real-time query or not.

Ie. Do you want to have access almost immediately to the data that the user stored in MySQL?

If so, I'd say that using an ETL like logstash is not the good tool for that.  
I shared my thoughts in this post: [http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/](http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/)

But if you are ok with let say 5 minutes latency, then Logstash is good IMO.  
Just create a logstash pipeline with an input jdbc plugin and an output elasticsearch plugin which runs every 5 minutes (jdbc plugin parameters) and you should be OK.

If you have specific questions about building that please ask in #logstash channel which is better for that.

---

<div class="post-metadata">

**Author:** ![ankibalyan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankibalyan/32/28951_2.png) [@ankibalyan](https://discuss.elastic.co/u/ankibalyan)\
**Post date:** [March 20, 2018, 7:45am UTC](https://discuss.elastic.co/t/choosing-the-right-path-for-elasticsearch-logstash/124592/3 "2018-03-20T07:45:43Z")

</div>

Thank you David Pilato,  
Yes, I do not need to have real-time query, I also do not want my application to slow while updating to elasticsearch for each bit.

Probably `input jdbc plugin` should work for me.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 20, 2018, 8:03am UTC](https://discuss.elastic.co/t/choosing-the-right-path-for-elasticsearch-logstash/124592/4 "2018-03-20T08:03:05Z")

</div>

> I also do not want my application to slow while updating to elasticsearch for each bit

I don't think that will be the case IMO.  
Specifically if you index asynchronously in elasticsearch and don't block any thread.  
You can also write to something like Kafka then read Kafka with logstash.

Reading the database every 5 minutes will put some pressure on your database. Also if you model is complex (like split on multiple tables), it might be hard to write a single query to fetch your object.

On the other hand, when your application holds the full object in memory, it's efficient, immediate to serialize in JSON and send the object to elasticsearch.

My 2 cents.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 17, 2018, 8:03am UTC](https://discuss.elastic.co/t/choosing-the-right-path-for-elasticsearch-logstash/124592/5 "2018-04-17T08:03:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.