Facing input-jdbc redshift docker problem

Elastic Stack Logstash
1

Hi guys,

I'm asking for an advice and assistance of logstash gurus with jdbc-input plugin :slight_smile:

There is a such structure:

  • Logstash is running in docker container, and it's composed with docker-compose.
  • there is couple of share folders with configs, including redshift-jdbc driver, .last_run file, query.sql and pipeline.yml.

Pipeline:

input {
  jdbc {
    jdbc_driver_library => "/data/driver.jar"
    jdbc_driver_class => "com.amazon.redshift.jdbc.Driver"
    jdbc_connection_string => "jdbc:redshift://10.101.5.237:5671/main"
    jdbc_user => "user"
    jdbc_password => "password"
    statement_filepath  => "/data/query.sql"
    last_run_metadata_path => "/data/.last_run"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
  }
}
output {
    amazon_es {
        hosts => [ "instanceurl.es.amazonaws.com" ]
        region => "eu-west-1"
        index => "index"
        document_type => "document"
        document_id => "%{id}"
    }
}

Dockerfile:

FROM docker.elastic.co/logstash/logstash-oss:6.4.2
RUN logstash-plugin install logstash-input-jdbc logstash-output-amazon_es
USER root # For changing folder mode
RUN mkdir /data ; chown -R logstash:logstash /data ; chmod -R 757 /data

docker-compose.yml:

version: '3'

services:
 logstash:
    container_name: my_db_logstash
    build:
      context: .
    restart: always
    volumes:
      - ./config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./pipeline:/usr/share/logstash/pipeline:ro
      - ./data:/data:rw
    ports:
      - "5004:5000"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"

driver.jar is downloaded from amazon website:

wget https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.16.1027/RedshiftJDBC42-no-awssdk-1.2.16.1027.jar -O data/driver.jar

All code is also available in github

But it throws an errors:

 logstash |   Error: Java::JavaSql::SQLException: No suitable driver found for jdbc:redshift://10.101.5.237:5671/main
 logstash |   Exception: Sequel::DatabaseConnectionError
 logstash |   Stack: java.sql.DriverManager.getConnection(java/sql/DriverManager.java:689)
 logstash |   java.sql.DriverManager.getConnection(java/sql/DriverManager.java:247)

I have tried also Postgresql driver as it suitable for Redshift, but it doesn't work as well.
By the way - exactly the same structure, but for mysql jdbc driver works fine.

I have feeling I'm missing something about jdbc_lobrary and jdbc_class part.

2

Facing same issue

Sachin (Channa Mereya Lyrics)

3

I also opened an issue in github repo of this plugin, here is it.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.