# Real Time data ingestion in ES from Multiple Source

**URL:** <https://discuss.elastic.co/t/real-time-data-ingestion-in-es-from-multiple-source/138069>\
**Category:** Elasticsearch\
**Created:** [July 1, 2018, 2:48am UTC](https://discuss.elastic.co/t/real-time-data-ingestion-in-es-from-multiple-source/138069 "2018-07-01T02:48:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nandan\_bigdata](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@nandan\_bigdata](https://discuss.elastic.co/u/nandan_bigdata)\
**Post date:** [July 1, 2018, 2:48am UTC](https://discuss.elastic.co/t/real-time-data-ingestion-in-es-from-multiple-source/138069/1 "2018-07-01T02:48:55Z")

</div>

Hi Team,

I am working on data ingestion project.  
I have multiple data source as below:-

1. Mysql Data
2. CSV Data
3. XML Data
4. JSON Data
5. Log data
6. MongoDB data

As data is generating every time, so I want to make sure data should be ingestion in real time from all data sources.  
So, how to make it sure I can able to do data ingestion in the correct way.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![rocknroll](https://avatars.discourse-cdn.com/v4/letter/r/c6cbf5/32.png) [@rocknroll](https://discuss.elastic.co/u/rocknroll)\
**Post date:** [July 1, 2018, 11:29pm UTC](https://discuss.elastic.co/t/real-time-data-ingestion-in-es-from-multiple-source/138069/2 "2018-07-01T23:29:00Z")

</div>

I would consider using Logstash which has a number of input plugins to help get you started, e.g. beats plugin for file-based inputs, or jdbc plugin for DB connections  
Check the Logstash doco here [https://www.elastic.co/guide/en/logstash/current/input-plugins.html](https://www.elastic.co/guide/en/logstash/current/input-plugins.html)

Logstash also provides filter plugins to handle csv and xml data formats. See here [https://www.elastic.co/guide/en/logstash/current/filter-plugins.html](https://www.elastic.co/guide/en/logstash/current/filter-plugins.html)

And of course the elasticsearch output plugin makes getting your documents into ES.

Once you've defined your inputs and filters, consider specifying logstash pipelines to segregate your different data streams

---

<div class="post-metadata">

**Author:** ![nandan\_bigdata](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@nandan\_bigdata](https://discuss.elastic.co/u/nandan_bigdata)\
**Post date:** [July 2, 2018, 6:50am UTC](https://discuss.elastic.co/t/real-time-data-ingestion-in-es-from-multiple-source/138069/3 "2018-07-02T06:50:22Z")

</div>

Thanks for the reply.  
As much As I can understand.  
Steps will be like below:-

1. Elasticsearch Service should be in running status [always].
2. Logstash Service should be in running status [always].
3. Kibana Service should be in running status [always].

Once I will run the command as [run first time only]  
bin/logstash -f mypipeline.conf

At that time whatever data will be in input location, those data will index and after that new data will come automatically.

For example :-

1. In first run, logstash will ingest 100 records from table 1 [At that time 100 records will be there].
2. later 5 records will insert into table 1 and automatically logstash will ingest those new data into ES.

Is it ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 30, 2018, 6:50am UTC](https://discuss.elastic.co/t/real-time-data-ingestion-in-es-from-multiple-source/138069/4 "2018-07-30T06:50:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.