Randomly missing events

[Nasser]

Hi
i have one pipeline processing 16 .conf file
in each file i have 2 jdbc
the issue is sometimes randomly i see logstash miss some jdbc !
i check the logstash-plain.log but nothing abnormal !

i tired to update logstash to latest version but still the issue appear
please your help

[Tomo_M (Tomohiro Mitani)]

@Nasser

miss some jdbc

What does this mean? Logstash miss some rows in SQL?
Can you share the pipelien?

[Nasser]

thanks for your reply
The scenario is sometimes all input run, and sometimes some of the input run and the other input not
there is no pattern for input running

also if you can see from my dashboard there is gaps
there is no issue in the elastic, because the input didn't get triggered in the first place !

here one of my file

input {
  jdbc {
    jdbc_driver_library => "/opt/ojdbc7.jar"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    jdbc_connection_string => "jdbc:oracle:thin:@*:*/*"
    jdbc_user => "*"
              jdbc_password => "*"
    schedule => "* * * * *"
    statement => "select * from my table"

  }
}
##############################################
input {
jdbc {
    jdbc_driver_library => "/opt/ojdbc7.jar"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    jdbc_connection_string => "jdbc:oracle:thin:@*/*"
    jdbc_user => "*"
              jdbc_password => "*"
    schedule => "* * * * *"
    statement => "select * from my table"

  }
}

my pipeline file as default

- pipeline.id: my-pipeline_1
  path.config: "my config path"

i hope my issue clear

[Tomo_M (Tomohiro Mitani)]

Isn't it just because there is no data in the daytime histogram buckets? Does data in gaps exist in the SQL?

[Nasser]

no it's not exist in sql
if i run the sql query i will have a result
but the issue logstash didn't execute the sql query

[Tomo_M (Tomohiro Mitani)]

It is possibly something related to this topic: "Dealing With Large Result-sets". That SQL query returns all the rows?