# How can i pull data from hive to logstash

**URL:** <https://discuss.elastic.co/t/how-can-i-pull-data-from-hive-to-logstash/95703>\
**Category:** Logstash\
**Created:** [August 3, 2017, 11:32am UTC](https://discuss.elastic.co/t/how-can-i-pull-data-from-hive-to-logstash/95703 "2017-08-03T11:32:31Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sesaravanan](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@sesaravanan](https://discuss.elastic.co/u/sesaravanan)\
**Post date:** [August 3, 2017, 11:32am UTC](https://discuss.elastic.co/t/how-can-i-pull-data-from-hive-to-logstash/95703/1 "2017-08-03T11:32:31Z")

</div>

Hi,

I'm very new to Logstash. I have access to read data from database, so I decided to **pull data** from a **remote database** to logstash. The data size will be minimum 1TB per day.  
How can i do capacity planning in terms of storage/networking/cpu/memory?  
What are the best practices I have to follow?

Thanks,  
Saravanan

---

<div class="post-metadata">

**Author:** ![josephjohney](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@josephjohney](https://discuss.elastic.co/u/josephjohney)\
**Post date:** [August 3, 2017, 11:35am UTC](https://discuss.elastic.co/t/how-can-i-pull-data-from-hive-to-logstash/95703/2 "2017-08-03T11:35:22Z")

</div>

> [@sesaravanan](#):
>
> I’m very new to Logstash. I have access to read data from database, so I decided to pull data from a remote database to logstash. The data size will be minimum 1TB per day.

you can use the various database plugins in logstash .

for example jdbc[JDBC plugin](https://www.elastic.co/guide/en/logstash/5.5/plugins-inputs-jdbc.html)

---

<div class="post-metadata">

**Author:** ![sesaravanan](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@sesaravanan](https://discuss.elastic.co/u/sesaravanan)\
**Post date:** [August 3, 2017, 11:55am UTC](https://discuss.elastic.co/t/how-can-i-pull-data-from-hive-to-logstash/95703/3 "2017-08-03T11:55:13Z")

</div>

Thanks for the response joseph 🙂

yes I can use jdbc plugin to pull data but I want to do capacity planning for that.

---

<div class="post-metadata">

**Author:** ![sesaravanan](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@sesaravanan](https://discuss.elastic.co/u/sesaravanan)\
**Post date:** [August 4, 2017, 3:59am UTC](https://discuss.elastic.co/t/how-can-i-pull-data-from-hive-to-logstash/95703/4 "2017-08-04T03:59:00Z")

</div>

Is it possible to pull data TB's of data with one logstash instance ? or I need to setup clustering?. How clustering will work for pulling data from remote database to Logstash?

---

<div class="post-metadata">

**Author:** ![josephjohney](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@josephjohney](https://discuss.elastic.co/u/josephjohney)\
**Post date:** [August 4, 2017, 5:22am UTC](https://discuss.elastic.co/t/how-can-i-pull-data-from-hive-to-logstash/95703/5 "2017-08-04T05:22:28Z")

</div>

You might have to do the horizontal scaling of your logstash pipeline.

[scaling](https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html) might give you some generic thoughts on the ELK stack scaling.

For your case there will be slight modification on the inputs.

Also, Please check the JDBC streaming plugins, which might be useful for your case.

---

<div class="post-metadata">

**Author:** ![sesaravanan](https://avatars.discourse-cdn.com/v4/letter/s/ce73a5/32.png) [@sesaravanan](https://discuss.elastic.co/u/sesaravanan)\
**Post date:** [August 4, 2017, 5:32am UTC](https://discuss.elastic.co/t/how-can-i-pull-data-from-hive-to-logstash/95703/6 "2017-08-04T05:32:04Z")

</div>

okay..

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 1, 2017, 5:32am UTC](https://discuss.elastic.co/t/how-can-i-pull-data-from-hive-to-logstash/95703/7 "2017-09-01T05:32:08Z")

</div>

Closed on September 1, 2017, 5:32am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.