On linux server, i wanted to have ELK setup with MariaDB through Docker.
I want to setup the Elastic search data with indexes for the maria DB tables from scratch and as well as for incremental changes sync.
These are docker processes running on my server
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7a915c230e6f mariadb:10.4 "docker-entrypoint.s…" 11 days ago Up 11 days 3306/tcp, 0.0.0.0:3309->3309/tcp, :::3309->3309/tcp sem_mariadb
205acbe6e117 docker.elastic.co/kibana/kibana:7.9.3 "/usr/local/bin/dumb…" 11 days ago Up 11 days 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp sem_kibana
5f582d9756cc docker.elastic.co/elasticsearch/elasticsearch:7.9.3 "/tini -- /usr/local…" 11 days ago Up 11 days 9200/tcp, 9300/tcp sem_elasticsearch
When i start the logstash, i was getting this error
sem_logstash | Pipeline_id:from-scratch-pipeline
sem_logstash | Plugin: <LogStash::Inputs::Jdbc jdbc_user=>"root", jdbc_password=>, statement_filepath=>"/usr/share/logstash/config/queries/mariadb-scratch.sql", clean_run=>true, jdbc_driver_library=>"/usr/share/logstash/pipeline/mariadb-java-client-2.4.2.jar", jdbc_connection_string=>"jdbc:mariadb://localhost:3309/test_database", sql_log_level=>"debug", id=>"4ce7641f8234f6b2c0618ef2d020b4774d759ce77be017c475514b12f77b020e", jdbc_driver_class=>"Java::org.mariadb.jdbc.Driver", record_last_run=>false, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_13495cfd-1407-4aa5-be35-49f82908a0fe", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, plugin_timezone=>"utc", last_run_metadata_path=>"/usr/share/logstash/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", lowercase_column_names=>true, use_prepared_statements=>false>
sem_logstash | Error: unable to load /usr/share/logstash/pipeline/mariadb-java-client-2.4.2.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)
sem_logstash | Exception: LogStash::PluginLoadingError
sem_logstash | Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/plugin_mixins/jdbc/common.rb:47:in block in load_driver_jars' sem_logstash | org/jruby/RubyArray.java:1809:in each'
sem_logstash | /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/plugin_mixins/jdbc/common.rb:42:in load_driver_jars' sem_logstash | /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/plugin_mixins/jdbc/common.rb:25:in load_driver'
sem_logstash | /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/inputs/jdbc.rb:275:in run' sem_logstash | /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in inputworker'
sem_logstash | /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'
sem_logstash | [2022-09-27T06:11:12,967][ERROR][logstash.javapipeline ][from-scratch-pipeline][4ce7641f8234f6b2c0618ef2d020b4774d759ce77be017c475514b12f77b020e] A plugin had an unrecoverable error. Will restart this plugin.
File : mariadb-scratch.conf
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/pipeline/mariadb-java-client-2.4.2.jar"
jdbc_driver_class => "Java::org.mariadb.jdbc.Driver"
jdbc_connection_string => "jdbc:mariadb://localhost:3309/test_database"
jdbc_user => "root"
jdbc_password => "root"
sql_log_level => "debug" # Set Logstash logging level as this
clean_run => true # Set to true for indexing from scratch
record_last_run => false
statement_filepath => "/usr/share/logstash/config/queries/mariadb-scratch.sql"
}
}
output {
stdout { codec => rubydebug { metadata => true } }
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "mariadb-%{+YYYY.MM}"
}
}
Even i have tried to change this path and placed the file in a folder and pointed to it with owner logstash.
jdbc_driver_library => "/usr/share/logstash/pipeline/mariadb-java-client-2.4.2.jar"
But no luck.
Can someone help me on resolving this issue