Scheduling Logstash Failed


(Meghla Chakravorty) #1

I am running a logstash pipeline to read logs from all files in a directory. I want my logstash conf file to run every 30 minutes in a day. Is there any command which I have to write in the conf file. Below is the snippet from the logstash conf file. Am using schedule to test it every 2 seconds (later will change it to 30 mins) but it seems schedule is not working.

Any help..

`

input
{
file
{
path=> "/home/cloudera/hive-metastore.log.2018-03-24"
start_position=> beginning
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
}
schedule => '2 * * * *'

}

`


(Magnus Bäck) #2

Only some inputs have a schedule option and the file input is not one of them. Besides, you've placed the option outside the file input, i.e. in the input block itself.

I suggest you keep Logstash running and have it continuously monitor the files you're interested in.


(Meghla Chakravorty) #3

input
{
file
{
path=> "/home/cloudera/yarnlogs/*"
start_position=> beginning
sincedb_path => "dev/null"
ignore_older => 0
stat_interval => 5

Changing to this worked. I am now trying to use filebeat , but stuck at configuring the filter part of logstash config to filebeat


(Magnus Bäck) #4

sincedb_path => "dev/null"

/dev/null, not dev/null.

I am now trying to use filebeat , but stuck at configuring the filter part of logstash config to filebeat

If you want help you need to be more specific.


(Meghla Chakravorty) #5

yes , thanks!

I am trying to input my log files through file beat to logstash and then run this logstash in every 30 mins(or any particular time interval) . Is there any particular config changes I need to do in either logstash / filebeat?


(Magnus Bäck) #6

What does your configuration look like now? What problem are you experiencing?


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.