Automatic data ingestion in logstash

[Kanishk_Madan]

Hello community,
This thread is regarding the automate of ingestion of data in logstash. I have created a python script in which I have written the command for the logstash data ingestion. The python script is also automatic by using crontab . but when the crontab is automated during the interval of time there is no ingestion of data. Is it possible for the logstash to ingest data automatically ?

[leandrojmp (Leandro Pereira)]

Can you explain better what you are trying to do?

What does your python script do? What is your logstash pipeline? How are you starting logstash?

Normally logstash would run as a service.

[Kanishk_Madan]

basically the python script would implement the command for the conf fle . In the python script I have written this command
import os
import time
print("Running")
os.system("/usr/share/logstash/bin/logstash -f /etc/logstash/logstash-sample.conf")

This python script would run the command for the logstash .to run this python script I have used crontab that would run at particular interval of time like:
45 10 * * * /usr/bin/python3.8 /root/Desktop/ingest.py

This crontab would work at 10:45 which would run the python script .

And in the conf file:-
input {
file {
path => "/root/Desktop/Sample.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}
filter {
csv {
separator => ","

columns => ["date","time","places"]

}

}
output {
elasticsearch{
hosts => "http://0.0.0.0:9200/"
index => "port"
}
stdout{}
}

So basically, I am running a python script that will ingest the data in logstash automatically using crontab.

[leandrojmp (Leandro Pereira)]

If I understand correctly, you want to read the same file each day at 10:45.

First, your pipeline will not do that, the sincedb_path option is wrong.

Using sincedb_path => "NULL" is telling logstash to save the last position read in a file called NULL, what you want is sincedb_path => "/dev/null".

I'm also not sure that your python script is calling logstash correctly, but in any case, it will not stop the logstash process unless you kill it, I'm not sure if os.system has any timeout, so you would start a new logstash process each day.

There is also no need to use a python script to call a system process.

In your case it would be best to leave logstash running as a service and just change the file.

Or if you still want to run logstash each time, I would recommend to use the stdin input and cat the contents of the file.

Just use:

input {
    stdin {}
}

And put it directly in your crontab

45 10 * * * cat /root/Desktop/Sample.csv | /usr/share/logstash/bin/logstash -f /etc/logstash/logstash-sample.conf`

[Kanishk_Madan]

ok let me try then I will get back to you.

[Kanishk_Madan]

Yeah it worked thanks for your help!!