Loading MYSQL database into Elasticsearch using Logstash

[anon98066099]

I am trying to load a simple table using logstash into elasticsearch. But i am unable to do so as i am getting the below error:

C:\Users\IYE8288\Downloads\logstash-6.3.1\logstash-6.3.1\bin>logstash -f mysql.conf
Sending Logstash's logs to C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logs which is now configured via log4j2.properties
[2018-07-24T16:43:11,864][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-07-24T16:43:12,964][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.1"}
[2018-07-24T16:43:19,228][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/outputs/elasticssearch>, :path=>"logstash/outputs/elasticssearch", :type=>"output", :name=>"elasticssearch"}
[2018-07-24T16:43:19,280][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any output plugin named 'elasticssearch'. Are you sure this is correct? Trying to load the elasticssearch output plugin resulted in this error: no such file to load -- logstash/outputs/elasticssearch", :backtrace=>["C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/plugins/registry.rb:210:in lookup_pipeline_plugin'", "C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/plugin.rb:140:inlookup'", "C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/plugins/plugin_factory.rb:89:in plugin'", "C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:110:inplugin'", "(eval):12:in <eval>'", "org/jruby/RubyKernel.java:994:ineval'", "C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:82:in initialize'", "C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:167:ininitialize'", "C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/agent.rb:305:inblock in converge_state'"]}
[2018-07-24T16:43:20,168][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Request you to help me on this..

[warkolm (Mark Walkom)]

It'd help us if you could share your config.

[NerdSec (Nachiket)]

@anon98066099

LogStash::PluginLoadingError", :message=>"Couldn't find any output plugin named 'elasticssearch

Is this a typo? Elasticssearch?

Also, as mark said. Please post your config when you ask a question. Makes it easier for people to help.

[NerdSec (Nachiket)]

@anon98066099

elasticssearch {

There is a typo. Could you correct this and try again? What error do you see?

[NerdSec (Nachiket)]

@anon98066099

"reason"=>"Invalid index name [Cities], must be lowercase",

This is the issue.

Also, note that document_type in deprecated in Elasticsearch 6.

@anon98066099

Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class is com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.

Have a look at this line as well.

[NerdSec (Nachiket)]

@anon98066099

Wed Jul 25 08:45:42 CDT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

This seems to be the issue. Are you using SSL?

[Jevgenij]

Nope, believe it's not. Seeing it all the time. It just warns the SSL settings should be explicitly set and for other related details. It works fine just like that until you actually use an SSL connection.

You can try

stdout { codec => json_lines }

or

stdout { codec => rubydebug }

in output section of your config file to see actual data being sent to Elastcisearch.