Facing Error while connecting Mariadb with elk through jdbc plugin

[Karthik_Yadav]

Logstash configuration file

input {
jdbc {
jdbc_validate_connection => true
jdbc_driver_library => "/etc/java-libs/mariadb-java-client-2.4.2.jar"
jdbc_driver_class => "Java::org.mariadb.jdbc.Driver"
jdbc_connection_string => "jdbc:mariadb://192.168.43.86:3306/test_database"
jdbc_user => "root"
jdbc_password => "mypass"
schedule => "*/1 * * * *"
statement => "SELECT * from logs"
sql_log_level => "debug"
record_last_run => true
last_run_metadata_path => "/etc/logstash/jdbc_last_run_metadata_path/mariadb"
}
}

output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "mariadb-%{+YYYY.MM}"
}
}

Error

Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>"root", schedule=>"*/1 * * * *", last_run_metadata_path=>"/etc/logstash/jdbc_last_run_metadata_path/mariadb", jdbc_validate_connection=>true, jdbc_password=>, statement=>"SELECT * from logs", jdbc_driver_library=>"/etc/java-libs/mariadb-java-client-2.4.2.jar", jdbc_connection_string=>"jdbc:mariadb://192.168.43.86:3306/test_database", sql_log_level=>"debug", id=>"d87f5594d1055b9a122011e290769739601a307a1b549475fac60b43de8cc3a8", jdbc_driver_class=>"Java::org.mariadb.jdbc.Driver", record_last_run=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_ac92da5f-7dca-43be-ade0-1c27c98fc3cc", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, plugin_timezone=>"utc", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, lowercase_column_names=>true, use_prepared_statements=>false>
Error: unable to load /etc/java-libs/mariadb-java-client-2.4.2.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)
Exception: LogStash::PluginLoadingError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:172:in block in load_driver_jars' org/jruby/RubyArray.java:1814:in each'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:167:in load_driver_jars' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:148:in load_driver'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.1/lib/logstash/inputs/jdbc.rb:273:in run' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:328:in inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:320:in `block in start_input'
[2020-05-19T14:09:32,037][ERROR][logstash.javapipeline ][main] A plugin had an unrecoverable error. Will restart this plugin.

Please help me with the proper solution

[Badger]

@Karthik_Yadav

Error: unable to load /etc/java-libs/mariadb-java-client-2.4.2.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)

logstash is unable to read that file. As it says, check permissions on the file and each directory in the path (you will need at least x permission on each directory).

[Karthik_Yadav]

Hi @Badger

I am facing the same issue after giving all permissions to each directory and file.

While checking with the curl --silent -XGET localhost:9200/_cat/indices/mariadb*?h=i,dc,
It is showing the following error

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/indices/mariadb*?h=i,dc]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/indices/mariadb*?h=i,dc]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

Can you please help with the proper solution @Badger

[ptamba]

@Karthik_Yadav

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/indices/mariadb*?h=i,dc]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/indices/mariadb*?h=i,dc]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

this is authentication error. your es expect auth credentials

[Karthik_Yadav]

Hi Sir,
I gave correct es credentials and gave full permission on each directory as mentioned in the above previous error, I am getting the same error

This is my logstash configuration file

These are the logs from logstash

Please help me with some commands and a proper solution.

[ptamba]

please avoid using screenshot with error messages. best to copy paste them and wrap with </> icon.

your error messages says “unable to load the driver”. the user running logstash needs permission to access the jdbc library