From mysql to aws elasticsearch using logstash

Elastic Stack Logstash
1

Hi everyone,

I'm new to ELK and I have never used AWS Elasticsearch before.

I'm trying to migrate data from mysql server to aws elasticsearch using logstash.

However, I got an error saying something like this:

[ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::OutputDelegator:0x3ad80dbe>", :error=>"undefined method `credentials' for nil:NilClass", :thread=>"#<Thread:0x1cddccdd run>"}
[ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `credentials' for nil:NilClass>,
[ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Stop/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method `unlock' for nil:NilClass"

I installed aws logstash plugin using:

bin/logstash-plugin install logstash-output-amazon_es

And my config file is the following:

input {
  jdbc {
    jdbc_driver_library => "/library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/mysql-connector-java-8.0.13.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql:SOMETHING"
    jdbc_user => "USER"
    jdbc_password => "PASSWORD"
    statement => "select * from TABLE_NAME"

      }
    }

output {
  amazon_es {
    hosts => ["MY_AWS_ES_DOMAIN"]
    region => "REGION"
    index => "INDEX_NAME"
    document_type => "TYPE"
    }
  stdout {
    codec => rubydebug
  }
}

Please help me find the way out from this :'(( Thank you so much in advance!

Thanks, Sam

2

In the case of elasticsearch, we will load data from mysql DB as shown below,

input {

jdbc {
jdbc_driver_library => "D:/Softwares/logstash-6.4.2/lib/com.mysql.jdbc_5.1.5.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/test"
jdbc_user => "root"
jdbc_password => "root"
statement => "SELECT * FROM sample"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
}

}

output{
elasticsearch { codec => json hosts => ["localhost:9200"] index => "mysqldata" }
stdout { codec => rubydebug }
}

3

Hi balumurari1,

Thank you for the reply.
I already successfully moved my data to my local elasticsearch server. However I'm having a problem with moving the data to aws elasticsearch..
Can you please help me with this?

thanks,
sam

4

Hello,

can you try this,added SSL port 443 with normal elasticsearch plugin as shown below

output {
elasticsearch {
hosts => ["test-xxxxxxxxxxxx-xxxx-1.es.amazonaws.com:443 "]
ssl => true
index => "index-test-1"
document_type => "type1"
}
}

5

Hi balumurari1,

I solved the issue by changing the output as:

output {
elasticsearch {
hosts => ["............es.amazonaws.com:80"]
index => "index_name"
}
}

Thank you anyway!

6

Great cheers,
Mark your answer as solution.

Regards,
Balu

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.