Connecting MySql to Logstash Using JDBC

Elastic Stack Logstash
1

Hi ,
this is my first time working on logstash ,
i am trying to send data from mysql to elasticsearch using logstash ,
but as a first step i specify json_line output
This is my output when running /usr/share/logstash/bin/logstash -f "/etc/logstash/logstash.conf" "--debug"
i really don't know where is the problem , can someone help me

DEBUG] 2020-05-13 10:23:37.476 [Ruby-0-Thread-6: :1] CompiledPipeline - Compiled output
P[output-stdout{"codec"=>"json_lines"}|[file]/etc/logstash/logstash.conf:12:3:```
stdout { codec => json_lines }

 into
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@c99a0cea
[DEBUG] 2020-05-13 10:23:37.475 [Ruby-0-Thread-5: :1] CompiledPipeline - Compiled output
 P[output-stdout{"codec"=>"json_lines"}|[file]/etc/logstash/logstash.conf:12:3:```
stdout {  codec => json_lines  }
```]
 into
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@c99a0cea
[INFO ] 2020-05-13 10:23:38.113 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[DEBUG] 2020-05-13 10:23:38.132 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.
[DEBUG] 2020-05-13 10:23:38.148 [Converge PipelineAction::Create<main>] javapipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x375d60b sleep>"}
[INFO ] 2020-05-13 10:23:38.316 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[DEBUG] 2020-05-13 10:23:38.408 [Api Webserver] agent - Starting puma
[DEBUG] 2020-05-13 10:23:38.425 [Api Webserver] agent - Trying to start WebServer {:port=>9600}
[DEBUG] 2020-05-13 10:23:38.598 [Api Webserver] service - [api-service] start
[INFO ] 2020-05-13 10:23:39.161 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[DEBUG] 2020-05-13 10:23:39.406 [[main]<jdbc] jdbc - loading /root/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[INFO ] 2020-05-13 10:23:40.950 [[main]<jdbc] jdbc - (0.037522s) SELECT version()
[INFO ] 2020-05-13 10:23:41.024 [[main]<jdbc] jdbc - (0.001793s) SELECT version()
[INFO ] 2020-05-13 10:23:41.354 [[main]<jdbc] jdbc - (0.008673s) SELECT count(*) AS `count` FROM (SELECT * FROM  zen_orders) AS `t1` LIMIT 1
[INFO ] 2020-05-13 10:23:41.374 [[main]<jdbc] jdbc - (0.007135s) SELECT count(*) AS `count` FROM (SELECT * FROM  zen_orders) AS `t1` LIMIT 1
[DEBUG] 2020-05-13 10:23:41.377 [[main]<jdbc] jdbc - Executing JDBC query {:statement=>"SELECT * FROM  zen_orders", :parameters=>{:sql_last_value=>1970-01-01 00:00:00 UTC}, :count=>23285}
[INFO ] 2020-05-13 10:23:41.738 [[main]<jdbc] jdbc - (0.338735s) SELECT * FROM  zen_orders
[WARN ] 2020-05-13 10:23:41.836 [[main]<jdbc] jdbc - Exception when executing JDBC query {:exception=>#<Sequel::DatabaseError: Java::JavaSql::SQLException: Zero date value prohibited>}
[DEBUG] 2020-05-13 10:23:42.056 [[main]-pipeline-manager] javapipeline - Input plugins stopped! Will shutdown filter/output workers. {:pipeline_id=>"main", :thread=>"#<Thread:0x375d60b run>"}
[DEBUG] 2020-05-13 10:23:42.088 [pool-4-thread-1] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2020-05-13 10:23:42.089 [pool-4-thread-1] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2020-05-13 10:23:42.090 [[main]-pipeline-manager] javapipeline - Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<Thread:0x5d03e81d run>"}
[DEBUG] 2020-05-13 10:23:42.144 [[main]-pipeline-manager] javapipeline - Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<Thread:0x2325ffcd dead>"}
[DEBUG] 2020-05-13 10:23:42.155 [[main]-pipeline-manager] stdout - Closing {:plugin=>"LogStash::Outputs::Stdout"}
[DEBUG] 2020-05-13 10:23:42.163 [[main]-pipeline-manager] pluginmetadata - Removing metadata for plugin 29f0ba7006a5f44a67f5942509b82c32830c981b461548ba2938dade7e2b4367
[DEBUG] 2020-05-13 10:23:42.166 [[main]-pipeline-manager] javapipeline - Pipeline has been shutdown {:pipeline_id=>"main", :thread=>"#<Thread:0x375d60b run>"}
[DEBUG] 2020-05-13 10:23:42.449 [LogStash::Runner] os - Stopping
[DEBUG] 2020-05-13 10:23:42.470 [LogStash::Runner] jvm - Stopping
[DEBUG] 2020-05-13 10:23:42.472 [LogStash::Runner] persistentqueue - Stopping
[DEBUG] 2020-05-13 10:23:42.472 [LogStash::Runner] deadletterqueue - Stopping
[DEBUG] 2020-05-13 10:23:42.482 [LogStash::Runner] agent - Shutting down all pipelines {:pipelines_count=>0}
[DEBUG] 2020-05-13 10:23:42.486 [LogStash::Runner] agent - Converging pipelines state {:actions_count=>0}
[INFO ] 2020-05-13 10:23:42.582 [LogStash::Runner] runner - Logstash shut down.
2

you have error in your sql statement. please provide your sql statement and config file to see how you’re using sql_last_value

3

thank you very much for your response ,
my error was that i have in my database 0000-00-00 00:00:00 dates , i delete it , and it works :smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.