# From mysql to aws elasticsearch using logstash

**URL:** <https://discuss.elastic.co/t/from-mysql-to-aws-elasticsearch-using-logstash/157627>\
**Category:** Logstash\
**Created:** [November 21, 2018, 6:01am UTC](https://discuss.elastic.co/t/from-mysql-to-aws-elasticsearch-using-logstash/157627 "2018-11-21T06:01:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![tripstore](https://avatars.discourse-cdn.com/v4/letter/t/8e8cbc/32.png) [@tripstore](https://discuss.elastic.co/u/tripstore)\
**Post date:** [November 21, 2018, 6:01am UTC](https://discuss.elastic.co/t/from-mysql-to-aws-elasticsearch-using-logstash/157627/1 "2018-11-21T06:01:16Z")

</div>

Hi everyone,

I'm new to ELK and I have never used AWS Elasticsearch before.

I'm trying to migrate data from mysql server to aws elasticsearch using logstash.

However, I got an error saying something like this:

```auto
[ERROR][logstash.pipeline] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::OutputDelegator:0x3ad80dbe>", :error=>"undefined method `credentials' for nil:NilClass", :thread=>"#<Thread:0x1cddccdd run>"}
[ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `credentials' for nil:NilClass>,
[ERROR][logstash.agent] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Stop/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method `unlock' for nil:NilClass"

```

I installed aws logstash plugin using:

```auto
bin/logstash-plugin install logstash-output-amazon_es

```

And my config file is the following:

```auto
input {
  jdbc {
    jdbc_driver_library => "/library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/mysql-connector-java-8.0.13.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql:SOMETHING"
    jdbc_user => "USER"
    jdbc_password => "PASSWORD"
    statement => "select * from TABLE_NAME"

      }
    }

output {
  amazon_es {
    hosts => ["MY_AWS_ES_DOMAIN"]
    region => "REGION"
    index => "INDEX_NAME"
    document_type => "TYPE"
    }
  stdout {
    codec => rubydebug
  }
}

```

Please help me find the way out from this :'(( Thank you so much in advance!

Thanks, Sam

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [November 21, 2018, 6:07am UTC](https://discuss.elastic.co/t/from-mysql-to-aws-elasticsearch-using-logstash/157627/2 "2018-11-21T06:07:23Z")

</div>

In the case of elasticsearch, we will load data from mysql DB as shown below,

input {

jdbc {  
jdbc\_driver\_library =\> "D:/Softwares/logstash-6.4.2/lib/com.mysql.jdbc\_5.1.5.jar"  
jdbc\_driver\_class =\> "com.mysql.jdbc.Driver"  
jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/test"  
jdbc\_user =\> "root"  
jdbc\_password =\> "root"  
statement =\> "SELECT \* FROM sample"  
jdbc\_paging\_enabled =\> "true"  
jdbc\_page\_size =\> "50000"  
}

}

output{  
elasticsearch { codec =\> json hosts =\> ["localhost:9200"] index =\> "mysqldata" }  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![tripstore](https://avatars.discourse-cdn.com/v4/letter/t/8e8cbc/32.png) [@tripstore](https://discuss.elastic.co/u/tripstore)\
**Post date:** [November 21, 2018, 6:39am UTC](https://discuss.elastic.co/t/from-mysql-to-aws-elasticsearch-using-logstash/157627/3 "2018-11-21T06:39:46Z")

</div>

Hi balumurari1,

Thank you for the reply.  
I already successfully moved my data to my local elasticsearch server. However I'm having a problem with moving the data to aws elasticsearch..  
Can you please help me with this?

thanks,  
sam

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [November 21, 2018, 6:47am UTC](https://discuss.elastic.co/t/from-mysql-to-aws-elasticsearch-using-logstash/157627/4 "2018-11-21T06:47:30Z")

</div>

Hello,

can you try this,added SSL port 443 with normal elasticsearch plugin as shown below

output {  
elasticsearch {  
hosts =\> ["[test-xxxxxxxxxxxx-xxxx-1.es.amazonaws.com:443](http://test-xxxxxxxxxxxx-xxxx-1.es.amazonaws.com:443/)"]  
ssl =\> true  
index =\> "index-test-1"  
document\_type =\> "type1"  
}  
}

---

<div class="post-metadata">

**Author:** ![tripstore](https://avatars.discourse-cdn.com/v4/letter/t/8e8cbc/32.png) [@tripstore](https://discuss.elastic.co/u/tripstore)\
**Post date:** [November 22, 2018, 1:09am UTC](https://discuss.elastic.co/t/from-mysql-to-aws-elasticsearch-using-logstash/157627/5 "2018-11-22T01:09:03Z")

</div>

Hi balumurari1,

I solved the issue by changing the output as:

output {  
elasticsearch {  
hosts =\> ["............es.amazonaws.com:80"]  
index =\> "index\_name"  
}  
}

Thank you anyway!

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [November 22, 2018, 5:12am UTC](https://discuss.elastic.co/t/from-mysql-to-aws-elasticsearch-using-logstash/157627/6 "2018-11-22T05:12:25Z")

</div>

Great cheers,  
Mark your answer as solution.

Regards,  
Balu

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 20, 2018, 5:12am UTC](https://discuss.elastic.co/t/from-mysql-to-aws-elasticsearch-using-logstash/157627/7 "2018-12-20T05:12:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.