How to identify and track logstash running or not at scheduled timeframe


(Suman Behara) #1

Hi All,

I have a requirement to identify logstash scheduler running or not at scheduled time. My application should know which scheduler are running , which are not running. current my schedulers are running fine with multi pipeline configuration.

In my logstash configuration we have multiple jdbc inputs and elastic output, each output has different
index name. we scheduled with different timeframe for all jdbc inputs.

Question : How to identify and track whether scheduler is running or not
daily perfectly with their respective scheduled time and how to identify which index data executed? Is there any where logstash maintains executed index name and executed timestamp. If fails to execute on scheduled time due to some reason, how to identify ,schedulers not runs on particular time.

here is sample logstash config file. Please suggest me what is the best approach to find out solution.

 	input { 
 jdbc
 {
 jdbc_connection_string=>"jdbc:db2://localhost:50050/DB"
 jdbc_user=>"admin"
 jdbc_password=>"admin"
 jdbc_driver_library=>"/usr/share/logstash/lib/db2jcc.jar"
 jdbc_driver_class=>"com.ibm.db2.jcc.DB2Driver"
 statement=>"select 'PRODUCT _Count' as name ,COUNT (DISTINCT PRODUCT_FKEY) as count from product"
 schedule=>"30 12,1 * * *"
 type=>"dsportal-product"
 }

 jdbc
 {
 jdbc_connection_string=>"jdbc:db2://localhost:50050/DB"
 jdbc_user=>"admin"
 jdbc_password=>"admin"
 jdbc_driver_library=>"/usr/share/logstash/lib/db2jcc.jar"
 jdbc_driver_class=>"com.ibm.db2.jcc.DB2Driver"
 statement=>"select count(*) as COUNT, current_date as Current_Date from upc"
 schedule=>"30 12,1 * * *"
 type=>"dsportal-upc"
}
 }
output {
if [type]=="dsportal-upc"
 { elasticsearch {
 hosts=>"localhost:9200"
 action=>"index"
 index=>"dsportal-product-%{+YYYY-MM-dd}"
 }
}
if [type]=="dsportal-product"
 { elasticsearch {
 hosts=>"localhost:9200"
 action=>"index"
 index=>"dsportal-upc-%{+YYYY-MM-dd}"
 }
}

}


(Magnus B├Ąck) #2

You could use a statsd or http output that posts to a heartbeat monitor like Lovebeat that can alert when the heartbeats stop arriving.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.