JDBC input plugin


(Vikas) #1

I am using below configuration in jdbc input:

Use_column_value = true
Tracking_column = end_dt (timestamp field)

I have some records in table where end_dt value is null showing that job is in running status. But when I run the config file it does not pick records where value is null as jdbc_last_run contains timestamp value.

Status will be updated for these records when end_dt value will be updated instead of null.

Please help how can I pick null value records showing running jobs and also update the status later on when end_dt value is updated.


(Guy Boertje) #2

You probably need two jdbc inputs:

  1. having WHERE end_dt IS NULL and type => end_dt_unset
  2. having WHERE end_dt > :sql_last_value ORDER BY end_dt and type => end_dt_set

Are you using Elasticsearch?

If so, have a conditional for one es output that bulk inserts end_dt_unset events with a doc_id and another conditional for another es output that updates the doc by id.

outputs {
  if [type] == "end_dt_unset" {

  }
  if [type] == "end_dt_set" {

  }
}

Do not put the two jdbc inputs on the same schedule. Perhaps put the NULL one on 15,30,45 seconds and the not NULL one on each minute at the 0 second mark. This gives LS/ES time to to index and make searchable the NULL ones.

From an ES mapping point of view, for the NULL doc cases, in the SELECT clause, set the end_dt to a timestamp literal of your choosing.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.