# Automatic data ingestion in logstash **URL:** \ **Category:** Logstash\ **Created:** [June 6, 2021, 2:37pm UTC](https://discuss.elastic.co/t/automatic-data-ingestion-in-logstash/275071 "2021-06-06T14:37:31Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Kanishk\_Madan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kanishk_madan/32/78928_2.png) [@Kanishk\_Madan](https://discuss.elastic.co/u/Kanishk_Madan)\ **Post date:** [June 6, 2021, 2:37pm UTC](https://discuss.elastic.co/t/automatic-data-ingestion-in-logstash/275071/1 "2021-06-06T14:37:31Z")
Hello community, This thread is regarding the automate of ingestion of data in logstash. I have created a python script in which I have written the command for the logstash data ingestion. The python script is also automatic by using crontab . but when the crontab is automated during the interval of time there is no ingestion of data. Is it possible for the logstash to ingest data automatically ? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 6, 2021, 3:04pm UTC](https://discuss.elastic.co/t/automatic-data-ingestion-in-logstash/275071/2 "2021-06-06T15:04:28Z")
Can you explain better what you are trying to do? What does your python script do? What is your logstash pipeline? How are you starting logstash? Normally logstash would run as a service. ---
**Author:** ![Kanishk\_Madan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kanishk_madan/32/78928_2.png) [@Kanishk\_Madan](https://discuss.elastic.co/u/Kanishk_Madan)\ **Post date:** [June 7, 2021, 5:31am UTC](https://discuss.elastic.co/t/automatic-data-ingestion-in-logstash/275071/3 "2021-06-07T05:31:05Z")
basically the python script would implement the command for the conf fle . In the python script I have written this command import os import time print("Running") os.system("/usr/share/logstash/bin/logstash -f /etc/logstash/logstash-sample.conf") This python script would run the command for the logstash .to run this python script I have used crontab that would run at particular interval of time like: 45 10 \* \* \* /usr/bin/python3.8 /root/Desktop/ingest.py This crontab would work at 10:45 which would run the python script . And in the conf file:- input { file { path =\> "/root/Desktop/Sample.csv" start\_position =\> "beginning" sincedb\_path =\> "NULL" } } filter { csv { separator =\> "," columns =\> ["date","time","places"] ``` } ``` } output { elasticsearch{ hosts =\> "[http://0.0.0.0:9200/](http://0.0.0.0:9200/)" index =\> "port" } stdout{} } So basically, I am running a python script that will ingest the data in logstash automatically using crontab. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 7, 2021, 2:24pm UTC](https://discuss.elastic.co/t/automatic-data-ingestion-in-logstash/275071/4 "2021-06-07T14:24:21Z")
If I understand correctly, you want to read the same file each day at 10:45. First, your pipeline will not do that, the `sincedb_path` option is wrong. Using `sincedb_path => "NULL"` is telling logstash to save the last position read in a file called `NULL`, what you want is `sincedb_path => "/dev/null"`. I'm also not sure that your python script is calling logstash correctly, but in any case, it will not stop the logstash process unless you kill it, I'm not sure if `os.system` has any timeout, so you would start a new logstash process each day. There is also no need to use a python script to call a system process. In your case it would be best to leave logstash running as a service and just change the file. Or if you still want to run logstash each time, I would recommend to use the `stdin` input and `cat` the contents of the file. Just use: ```auto input { stdin {} } ``` And put it directly in your crontab ```auto 45 10 * * * cat /root/Desktop/Sample.csv | /usr/share/logstash/bin/logstash -f /etc/logstash/logstash-sample.conf` ``` ---
**Author:** ![Kanishk\_Madan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kanishk_madan/32/78928_2.png) [@Kanishk\_Madan](https://discuss.elastic.co/u/Kanishk_Madan)\ **Post date:** [June 7, 2021, 2:40pm UTC](https://discuss.elastic.co/t/automatic-data-ingestion-in-logstash/275071/5 "2021-06-07T14:40:34Z")
ok let me try then I will get back to you. ---
**Author:** ![Kanishk\_Madan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kanishk_madan/32/78928_2.png) [@Kanishk\_Madan](https://discuss.elastic.co/u/Kanishk_Madan)\ **Post date:** [June 9, 2021, 8:44am UTC](https://discuss.elastic.co/t/automatic-data-ingestion-in-logstash/275071/6 "2021-06-09T08:44:00Z")
Yeah it worked thanks for your help!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 7, 2021, 8:44am UTC](https://discuss.elastic.co/t/automatic-data-ingestion-in-logstash/275071/7 "2021-07-07T08:44:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.