Logstash Scheduler

Elastic Stack Logstash
1

Hey Elasticworld,

I'm current Logstash version 7.9 ,I have a encounter a problem in Logstash, in which have a 4 pipeline which are using "JBDC" driver for inserting data into Elasticsearch. **Now I have to add another pipeline through which I could insert the data in Elasticsearch by a "CSV" on a regular basis. **
I have configured the config file it Run perfectly , but I have to Schedule it at once a day at 1 am every day along with other Scheduler which are set on a different time (for example 3 am and 4am) Can anyone help me out, How can I add a Scheduler in the following added config.

I have seen your post regarding this issue.

[type or paste code here](https://discuss.elastic.co/t/scheduling-a-logsatsh-config-file/67831/4)

input {
  file {
    type => "CSV"
    path => "C:\Users\Administrator\Desktop\abc\**.csv"
    start_position => "beginning"
   sincedb_path => "/dev/null"
    close_older =>"10 second"
    exit_after_read => true
    mode => "read"

  }
}
filter {
  csv {
      separator => ","
  }
}


output {
    elasticsearch {
      hosts => ["http:XXXXXXX:9200"]
    index => ["ABC"]
      user => "XXXXX"
      password =>"XXXXX"
  }
stdout {}
}
2

The file input doesn't have a schedule, it'll always watch the pattern you define and watch for new files.

Will that not work for you?

3

Hey @warkolm, thank you for your response.

Yes it always watch for a new file but if this process remain open then other corn job would not run as a logstash file is still in use.

I hope my explanation is understandable.

4

If you're already using cron, then just use it for this.
Change the file input to stdin and then cat the file in.

5

@warkolm by cron I mean other Scheduler place in different config
which are being called by pipeline.yml ( etc/logstash)
Now I have to add a new config to pipeline.yml in which I can read a CSV from a file location mention above on a daily basis and save it to Elasticsearch.

Secondly
Can we apply CSV filter in stdin?

I have tired it out while the logstash is running as service but it fails and give a error that another logstash instance is in used.

$ cat file | ./bin/logstash -f /bin/logstashmyconfig.conf
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.