Migration data from MariaDB to Elasticsearch via Logstash

[Vladpov]

I'm trying to migrate data from MariaDB to Elasticsearch by using Logstash.

I never worked with jdbc before so I installed jdbc with sudo apt-get install libmysql-java command and created the following mysql.conf file in logstash conf.d directory:

input {
  jdbc {
    jdbc_driver_library => "/usr/share/java/mysql-connector-java-5.1.45.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/ecomdb"
    jdbc_user => user12
    jdbc_password => user
    tracking_column => "regdate"
    use_column_value =>true
    statement => "SELECT * FROM ecomdb.customer where regdate >:sql_last_value;"
    schedule => "*/5 * * * * *"
  }
}
output {
  elasticsearch {
    document_id => "%{id}"
    document_type => "doc"
    index => "test2"
    hosts => ["http://localhost:9200"]
  }
  stdout{
  codec => rubydebug
  }
}

After I run logstash it shows me the following ERROR message:

logstash[14954]: { 2092 rufus-scheduler intercepted an error:
logstash[14954]:   2092   job:
logstash[14954]:   2092     Rufus::Scheduler::CronJob "*/5 * * * * *" {}
logstash[14954]:   2092   error:
logstash[14954]:   2092     2092
logstash[14954]:   2092     LogStash::PluginLoadingError
logstash[14954]:   2092     Unable to find driver class via URLClassLoader in given driver jars: com.mysql.jdbc.Driver and com.mysql.jdbc.Driver
logstash[14954]:   2092       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/plugin_mixins/jdbc/jdbc.rb:181:in `open_jdbc_connection'
logstash[14954]:   2092       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/plugin_mixins/jdbc/jdbc.rb:253:in `execute_statement'
logstash[14954]:   2092       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/inputs/jdbc.rb:309:in `execute_query'
logstash[14954]:   2092       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.16/lib/logstash/inputs/jdbc.rb:276:in `block in run'
logstash[14954]:   2092       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in `do_call'
logstash[14954]:   2092       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258:in `do_trigger'
logstash[14954]:   2092       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300:in `block in start_work_thread'
logstash[14954]:   2092       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299:in `block in start_work_thread'
logstash[14954]:   2092       org/jruby/RubyKernel.java:1425:in `loop'
logstash[14954]:   2092       /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `block in start_work_thread'

MariaDB documentation says : Please note that the driver class provided by MariaDB Connector/J is not com.mysql.jdbc.Driver but org.mariadb.jdbc.Driver! but when I changed driver class in my logstash config file it still doesn't work.

Could it be any problem with the way I installed MySQL jdbc? If so, how can I install MariaDB jdbc from this website?

Thanks in advance!!

[Vladpov]

I'd like to share my solution of the problem.

Current logstash config file:

input {
  jdbc {
    jdbc_validate_connection => true
    jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mariadb-java-client-2.4.2.jar"
    jdbc_driver_class => "Java::org.mariadb.jdbc.Driver"
    jdbc_connection_string => "jdbc:mariadb://localhost:3306/logdb"
    jdbc_user => "root"
    jdbc_password => "root"
    schedule => "* * * * * *"
    statement => "SELECT * FROM logs;"
  }
}

}
output {
  elasticsearch {
    document_id => "%{doc_id}"
    document_type => "doc"
    index => "test3"
    hosts => ["http://localhost:9200"]
  }
  stdout{
  codec => rubydebug
  }
}

The thing is that you have to download mariadb jdbc driver wget https://downloads.mariadb.com/Connectors/java/connector-java-2.4.2/mariadb-java-client-2.4.2.jar and move it to /usr/share/logstash/logstash-core/lib/jars/ (logstash installation directory).

Cheers!