Tracking column not found in dataset

Elastic Stack Logstash
1

Good morning,

I've an issue with the tracking_column with the mysql plug-in.

the conf.d

input {
  jdbc {
    jdbc_connection_string => "jdbc:mysql://localhost:3306/KPIS?useSSL=false"
    jdbc_user => "labo"
    jdbc_password => "arcsight_L4B0"
    jdbc_driver_library => "/<path connector>/mysql-connector-java-5.1.47.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    schedule => "10 7 * * 1,2,3,4,5,6,7"
    statement => "Select reglasArcsight.idreglasArcsight,    reglasArcsight.prioridad,    reglasArcsight.fecha,    reglasArcsight.nombreAgente,    reglasArcsight.flexString1,    reglasArcsight.flexString2,    reglasArcsight.regla,    reglasArcsight.nombre,    reglasArcsight.generatorName,    reglasArcsight.nombrePais,    reglasArcsight.codigoPais,    reglasArcsight.localizacion,    reglasArcsight.ipAtacante,    reglasArcsight.dispositivo,    reglasArcsight.nombreObjetivo,    reglasArcsight.usuarioObjetivo,    reglasArcsight.ipDestino,    reglasArcsight.nombreAtacante,    reglasArcsight.nombreDispositivo,    reglasArcsight.url,    reglasArcsight.fabricante,    reglasArcsight.nombreProducto FROM KPIS.reglasArcsight where idreglasArcsight>:sql_last_value"
    use_column_value => true
    clean_run => false
    tracking_column => idreglasArcsight
    jdbc_paging_enabled => true
    jdbc_page_size => 250000
    last_run_metadata_path => "<pathFile"
 }
}
output {
  elasticsearch {
  "hosts" => "localhost:9200"
  "index" => "reglas"
  }
stdout { codec => json_lines }
}

the error that shows logstash log is the following:
tracking_column not found in dataset. {:tracking_column=>"idreglasArcsight"}

any idea?

regards

1 Like
2

Make the trackingcolumn smallcaps

3

Agreed with @inhinyera16 - it could be a casing issue (c.f github issue). You are also using SELECT reglasArcsight.idreglasArcsight and it is possible that the tracking_column needs to contain the full name of the column (i.e. reglasArcsight.idreglasArcsight or reglasarcsight.idreglasarcsight depending on the lowercase_column_names value).

4

hi everybody,

I've changed the different values (name, case sensitive, etc) but the problem persists

regards

5

@legolas_bilbao . This is what I would recommend:

a) Check that the SQL statement syntax is correct and that the SQL statement runs successfully against your database? This is the SQL statement that you provided:

Select reglasArcsight.idreglasArcsight,    reglasArcsight.prioridad,    reglasArcsight.fecha,    reglasArcsight.nombreAgente,    reglasArcsight.flexString1,    reglasArcsight.flexString2,    reglasArcsight.regla,    reglasArcsight.nombre,    reglasArcsight.generatorName,    reglasArcsight.nombrePais,    reglasArcsight.codigoPais,    reglasArcsight.localizacion,    reglasArcsight.ipAtacante,    reglasArcsight.dispositivo,    reglasArcsight.nombreObjetivo,    reglasArcsight.usuarioObjetivo,    reglasArcsight.ipDestino,    reglasArcsight.nombreAtacante,    reglasArcsight.nombreDispositivo,    reglasArcsight.url,    reglasArcsight.fabricante,    reglasArcsight.nombreProducto FROM KPIS.reglasArcsight where idreglasArcsight>:sql_last_value

b) You can run logstash from the command line with the --debug flag. For example: bin/logstash --debug -f CONFIG_PATH. You can attach the debug logs here, that would help us troubleshooting the issue.

Thank you.

6

thanks, with the log I've seen the problem and i feel a bit stupid.

regards

7

You're welcome @legolas_bilbao. Feel free to share your feedback/solution with us :slightly_smiling_face: that could help someone else facing similar scenario.

8

I see in the log that the index is recieving in lowcases then I've changed the column to lowcases and its works.

many thanks for the tips

2 Likes
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.