JDBC input plugin

[Goyal.vikas87 (Vikas)]

I am using below configuration in jdbc input:

Use_column_value = true
Tracking_column = end_dt (timestamp field)

I have some records in table where end_dt value is null showing that job is in running status. But when I run the config file it does not pick records where value is null as jdbc_last_run contains timestamp value.

Status will be updated for these records when end_dt value will be updated instead of null.

Please help how can I pick null value records showing running jobs and also update the status later on when end_dt value is updated.

[guyboertje]

You probably need two jdbc inputs:

1. having WHERE end_dt IS NULL and type => end_dt_unset
2. having WHERE end_dt > :sql_last_value ORDER BY end_dt and type => end_dt_set

Are you using Elasticsearch?

If so, have a conditional for one es output that bulk inserts end_dt_unset events with a doc_id and another conditional for another es output that updates the doc by id.

outputs {
  if [type] == "end_dt_unset" {

  }
  if [type] == "end_dt_set" {

  }
}

Do not put the two jdbc inputs on the same schedule. Perhaps put the NULL one on 15,30,45 seconds and the not NULL one on each minute at the 0 second mark. This gives LS/ES time to to index and make searchable the NULL ones.

From an ES mapping point of view, for the NULL doc cases, in the SELECT clause, set the end_dt to a timestamp literal of your choosing.