Hello,
I'm new to logstash, I'm running a logstash pipeline configuration in an aws instance with the following configuration :
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://my.domain:3306/db_name"
jdbc_user => "user"
jdbc_password => "pass"
jdbc_driver_library => "/etc/mysql-connector/mysql-connector-java-8.0.27.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
schedule => "* * * * * *"
statement =>
"SELECT spi.id as product_id, spi.sid as site_id, spi.modified as modified
FROM site_products spi
WHERE product_id > :sql_last_value"
use_column_value => true
tracking_column => "product_id"
jdbc_paging_enabled => true
jdbc_page_size => 10000
jdbc_fetch_size => 1000
last_run_metadata_path => "/etc/logstash/sql_last_value.d/sql_last_value"
}
}
output {
elasticsearch {
"hosts" => "elk.domain:9200"
"index" => "my_index"
"document_id" => "%{product_id}"
}
stdout { codec => json_lines }
}
The configuration was running correctly, however, after a restart of the instance, it does not seem to transfer anything anymore, and seems to be stuck in the select version() part, without any logs about data transfer (which was the case before)
[2022-01-19T13:02:52,541][INFO ][logstash.javapipeline ][uk-pipeline] Starting pipeline {:pipeline_id=>"uk-pipeline", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/logstash_uk.conf"], :thread=>"#<Thread:0x29160f34 run>"}
[2022-01-19T13:02:53,447][INFO ][logstash.javapipeline ][uk-pipeline] Pipeline Java execution initialization time {"seconds"=>0.9}
[2022-01-19T13:02:53,527][INFO ][logstash.javapipeline ][uk-pipeline] Pipeline started {"pipeline.id"=>"uk-pipeline"}
[2022-01-19T13:02:53,614][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:"uk-pipeline"], :non_running_pipelines=>[]}
[2022-01-19T13:02:56,096][INFO ][logstash.inputs.jdbc ][uk-pipeline][ecca4a029451db93a831d53559da7bcf321bb296f91bffbb89535e2855bc79f7] (0.020264s) SELECT version()
[2022-01-19T13:02:56,126][INFO ][logstash.inputs.jdbc ][uk-pipeline][ecca4a029451db93a831d53559da7bcf321bb296f91bffbb89535e2855bc79f7] (0.001189s) SELECT version()
Do you have any idea of the origin of the issue ? and how we can debug it to have more details ?
Thanks !