# How to automate data ingestion with logstash pipelines?

**URL:** <https://discuss.elastic.co/t/how-to-automate-data-ingestion-with-logstash-pipelines/52029>\
**Category:** Logstash\
**Created:** [June 7, 2016, 7:31am UTC](https://discuss.elastic.co/t/how-to-automate-data-ingestion-with-logstash-pipelines/52029 "2016-06-07T07:31:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![qbadx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/qbadx/32/20862_2.png) [@qbadx](https://discuss.elastic.co/u/qbadx)\
**Post date:** [June 7, 2016, 7:31am UTC](https://discuss.elastic.co/t/how-to-automate-data-ingestion-with-logstash-pipelines/52029/1 "2016-06-07T07:31:02Z")

</div>

Hi there,

I'm using ELK in order to ingest, store and visualize data, no fancy things..

Everything is working fine but each time I have new data to ingest I have to execute manually the command /opt/logstash/bin/logstash -f mypipeline.conf

I was wondering how to automate this last step in order to ingest the data in elastisearch each time new data arrive in the inut folder defined in my pipeline conf file?

thx

---

<div class="post-metadata">

**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\
**Post date:** [June 7, 2016, 8:23am UTC](https://discuss.elastic.co/t/how-to-automate-data-ingestion-with-logstash-pipelines/52029/2 "2016-06-07T08:23:25Z")

</div>

What type of input plugin are you using for logstash?

> [@qbadx](#):
>
> in order to ingest the data in elastisearch each time new data arrive in the inut folder defined in my pipeline conf file

If I'm reading this correctly, you are probably using the file input plugin:

> **[File input plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html)**

It should be able to handle new incoming data automatically, so no need to restart logstash, just leave it running on its own and check for new data IIRC.

As an aside - you might have better luck getting an answer when asking in the logstash forum.

---

<div class="post-metadata">

**Author:** ![qbadx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/qbadx/32/20862_2.png) [@qbadx](https://discuss.elastic.co/u/qbadx)\
**Post date:** [June 7, 2016, 8:50am UTC](https://discuss.elastic.co/t/how-to-automate-data-ingestion-with-logstash-pipelines/52029/3 "2016-06-07T08:50:36Z")

</div>

Hello Isabel,

thx for the reply! I have moved the post to the logstash forum.

I'm indeed using the input plugin :

file {  
path =\> "/path/to/myfiles\*.csv"  
start\_position =\> beginning  
sincedb\_path =\> "/dev/null"  
}

I guess I'm missing an important option that would allow to check if new files are present or not..

"discover\_interval" or "stat\_interval" ? or the sincedb-path ??

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 10, 2016, 5:35am UTC](https://discuss.elastic.co/t/how-to-automate-data-ingestion-with-logstash-pipelines/52029/4 "2016-06-10T05:35:15Z")

</div>

Logstash should automatically discover new files within a few seconds (`discover_interval`).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:53am UTC](https://discuss.elastic.co/t/how-to-automate-data-ingestion-with-logstash-pipelines/52029/5 "2017-07-06T04:53:47Z")

</div>