How do we know which jar version to use with a particular version of java and logstash?
The reason I am asking is the following:
configuration file
input {
jdbc {
jdbc_driver_library => "/etc/logstash/mysql-connector-java-8.0.18.jar"
jdbc_connection_string => "jdbc:mysql://server:3306/database"
jdbc_user => "user"
jdbc_password => "password"
statement => "SELECT * FROM readings"
}
}
output {
elasticsearch {
index => "stats"
action => "index"
document_id => "%{readings_id}"
hosts => "http://localhost:9200"
}
}The error that is being produced is as follows
[root@elk logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/32-db.conf...
**Error: com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?**
Exception: LogStash::PluginLoadingError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/plugin_mixins/jdbc/jdbc.rb:190:in `open_jdbc_connection'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/plugin_mixins/jdbc/jdbc.rb:253:in `execute_statement'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/inputs/jdbc.rb:309:in `execute_query'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/inputs/jdbc.rb:281:in `run'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `block in start_input'
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[ERROR] 2019-11-08 04:03:22.121 [[main]<jdbc] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>"fish", jdbc_password=><password>, statement=>"SELECT * FROM readings", jdbc_driver_library=>"/etc/logstash/mysql-connector-jav a-8.0.18.jar", jdbc_connection_string=>"jdbc:mysql://[mikesdevhub.com:3306/fish](http://mikesdevhub.com:3306/fish)", id=>"293177a1017f2639f711c05bf3a6359e590927fe7b8704d60adb96c8c80a3c1d", jdbc_driver_class=> "com.mysql.jdbc.Driver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_52940cbb-e65e-472f-9f19-620b595bb66b", enable_metric=>true, charset=>"UTF-8">, jdbc _paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retr y_attempts=>1, connection_retry_attempts_wait_time=>0.5, plugin_timezone=>"utc", last_run_metadata_path=>"/root/.logstash_jdbc_last_run", use_column_value=>false, tracking_ column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true, use_prepared_statements=>false>
Error: com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
Exception: LogStash::PluginLoadingError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/plugin_mixins/jdbc/jdbc.rb:190:in `open_jdbc_connection'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/plugin_mixins/jdbc/jdbc.rb:253:in `execute_statement'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/inputs/jdbc.rb:309:in `execute_query'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/inputs/jdbc.rb:281:in `run'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `block in start_input'
^C[WARN ] 2019-11-08 04:03:22.375 [SIGINT handler] runner - SIGINT received. Shutting down.
^C[FATAL] 2019-11-08 04:03:22.562 [SIGINT handler] runner - SIGINT received. Terminating immediately..
^C[FATAL] 2019-11-08 04:03:22.690 [SIGINT handler] runner - SIGINT received. Terminating immediately..
[ERROR] 2019-11-08 04:03:22.810 [LogStash::Runner] Logstash - org.jruby.exceptions.ThreadKillI am running logstash 6.8. I have followed multiple guides which I have said the same thing which is what I have implemented.
Michael
I have many config running using jdbc. here is my config
as you can see I have driver_library not there or comment out.
I have copy ojdbc8.jar in to /usr/share/logstash/logstash-core/lib/jars/ directory
try that and I think it will work.
input {
jdbc {
jdbc_validate_connection => true
jdbc_connection_string => "jdbc:oracle:thin:@hostname1:1521/my_db_name"
jdbc_user => "logstash"
jdbc_password => "password_logstash"
# jdbc_driver_library => "/usr/lib/oracle/12.2/client64/lib/ojdbc8.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
statement_filepath=> "/etc/logstash/conf.d/sql/my_expensive_query.sql"
schedule => "*/5 * * * *"
clean_run=>true
}
}Thank you for your reply. I think it was a permission. I get the following output now
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-11-08 21:32:45.842 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-08 21:32:45.864 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.4"}
[INFO ] 2019-11-08 21:32:56.385 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-11-08 21:32:57.228 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2019-11-08 21:32:57.571 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2019-11-08 21:32:57.864 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2019-11-08 21:32:57.868 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2019-11-08 21:32:58.101 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[INFO ] 2019-11-08 21:32:58.156 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template
[INFO ] 2019-11-08 21:32:58.226 [Ruby-0-Thread-5: :1] elasticsearch - Index Lifecycle Management is set to 'auto', but will be disabled - Your Elasticsearch cluster is before 7.0.0, which is the minimum version required to automatically run Index Lifecycle Management
[INFO ] 2019-11-08 21:32:58.228 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[INFO ] 2019-11-08 21:32:58.381 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x133e215f run>"}
[INFO ] 2019-11-08 21:32:58.546 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-11-08 21:32:59.392 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[INFO ] 2019-11-08 21:33:01.588 [[main]<jdbc] jdbc - (0.079883s) SELECT * FROM readings
[INFO ] 2019-11-08 21:33:03.087 [[main]-pipeline-manager] pipeline - Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x133e215f run>"}
[INFO ] 2019-11-08 21:33:03.310 [LogStash::Runner] runner - Logstash shut down.My configuration file is below
input {
jdbc {
#jdbc_driver_library => "/etc/logstash/mysql-connector-java-8.0.18.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://mikesdevhub.com:3306/fish"
jdbc_user => "fish"
jdbc_password => "Fu59e123#"
statement => "SELECT * FROM readings"
}
}
output {
elasticsearch {
index => "fishstats"
action => "index"
document_id => "%{readings_id}"
hosts => "http://localhost:9200"
}
}Thank you I think it was part permissions. Things are working and things have been updated meaning it has pulled updated information from the database.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.