Elastic Search My SQL Integration via LogStash

[sg850r (Suman)]

We are trying to use Elastic Search for search capabilities for our site. All data in in MySQL and we need this data copied to elastic search via LogStash.copied here is the conf file and log file . But when I run the logstash command I get a error saying : SSLConnectionSocketFactory not found in packages
We have a hit a roadblock here and need pointers to overcome this error.
Thanks
Suman

config file:

file: simple-out.conf

input {
jdbc {
# MySQL jdbc connection string to our database, mydb
jdbc_connection_string => "jdbc:mysql://myip/mydatabase?useSSL=false"
# The user we wish to execute our statement as
jdbc_user => "my user"
jdbc_password => "my pwd"
# The path to our downloaded jdbc driver
jdbc_driver_library => "C:\JavaDevelopment\TomcatServer\apache-tomcat-8.5.20\lib\mysql-connector-java-5.1.45-bin.jar"
# The name of the driver class for Postgresql
jdbc_driver_class => "com.mysql.jdbc.Driver"
# our query
statement => "SELECT * from testtable"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
hosts => "http://localhost:9200"
index => "test-migrate"
document_type => "data"
ssl => "false"
}

}

Log file:
[2017-12-20T08:52:50,719][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NameError: SSLConnectionSocketFactory not found in packages org.apache.http.client.methods, org.apache.http.client.entity, org.apache.http.client.config, org.apache.http.config, org.apache.http.conn.socket, org.apache.http.impl, org.apache.http.impl.client, org.apache.http.impl.conn, org.apache.http.impl.auth, org.apache.http.entity, org.apache.http.message, org.apache.http.params, org.apache.http.protocol, org.apache.http.auth, java.util.concurrent, org.apache.http.client.protocol, org.apache.http.conn.ssl, java.security.cert, java.security.spec, java.security, org.apache.http.client.utils; last error: cannot load Java class org.apache.http.client.utils.SSLConnectionSocketFactory>, :backtrace=>["uri:classloader:/jruby/java/core_ext/module.rb:45:in const_missing'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/client.rb:603:inssl_socket_factory_from_options'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/client.rb:394:in pool_builder'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/client.rb:402:inpool'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/client.rb:208:in initialize'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:26:ininitialize'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:273:in build_adapter'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:277:inbuild_pool'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in initialize'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:101:increate_http_client'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:97:in build'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch.rb:230:inbuild_client'", "C:/JavaDevelopment/logstash-6.1.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:24:in register'", "C:/JavaDevelopment/logstash-6.1.0/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "C:/JavaDevelopment/logstash-6.1.0/logstash-core/lib/logstash/output_delegator.rb:43:in register'", "C:/JavaDevelopment/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:343:inregister_plugin'", "C:/JavaDevelopment/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:354:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "C:/JavaDevelopment/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:354:in register_plugins'", "C:/JavaDevelopment/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:743:inmaybe_setup_out_plugins'", "C:/JavaDevelopment/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:364:in start_workers'", "C:/JavaDevelopment/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:288:inrun'", "C:/JavaDevelopment/logstash-6.1.0/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"], :thread=>"#<Thread:0x1b103fad run>"}
[2017-12-20T08:52:50,777][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:main, action_result: false", :backtrace=>nil}

[arisbanach]

Might be a stupid question, but did you install the logstash-input-jdbc plugin?

[sg850r (Suman)]

the JDBC plugin that reads data from MySQL works perfectly. When I run logstash with the simple conf file to read data from MySQL and write to standard out it works. It doesnt work when using the elastic search plugin. It gives this cannot load Java class org.apache.http.client.utils.SSLConnectionSocketFactory ERROR.

[arisbanach]

ah you mean it doesn't work when using the logstash-output-elasticsearch plugin? and that is installed too, right? can you try putting the "hosts" value in square brackets like hosts => ["http://localhost:9200"]?

[sg850r (Suman)]

Yes I tried using hosts => ["http://localhost:9200"] ...but get same error . The logstash elastic search output plugin exists because it is able to read the parameters provided in elasticsearch {} output. I see the following error in log file:

[2017-12-20T14:25:08,269][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::OutputDelegator:0x4ad9024e @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0x173af7bf @metric=#<LogStash::Instrument::Metric:0x7c8cb53d @collector=#<LogStash::Instrument::Collector:0x2ab0d75d @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x28797b37 @store=#<Concurrent:0x00000000064380 entries=2 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x68bee688, @fast_lookup=#<Concurrent:0x00000000064384 entries=52 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"18be2952e4128a38734de3ec0ab20ca289c76aa0-2"]>, @metric=#<LogStash::Instrument::NamespacedMetric:0x5d24b8bd @metric=#<LogStash::Instrument::Metric:0x7c8cb53d @collector=#<LogStash::Instrument::Collector:0x2ab0d75d @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x28797b37 @store=#<Concurrent:0x00000000064380 entries=2 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x68bee688, @fast_lookup=#<Concurrent:0x00000000064384 entries=52 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#<LogStash::Logging::Logger:0x2172fb1f @logger=#Java::OrgApacheLoggingLog4jCore::Logger:0x6cebd103>, @out_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"18be2952e4128a38734de3ec0ab20ca289c76aa0-2", :events] key: out value: 0, @in_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"18be2952e4128a38734de3ec0ab20ca289c76aa0-2", :events] key: in value: 0, @strategy=#<LogStash::OutputDelegatorStrategies::Shared:0x3cf3ff1e @output=<LogStash::Outputs::ElasticSearch action=>"index", index=>"test-migrate", document_type=>"data", ssl=>false, id=>"18be2952e4128a38734de3ec0ab20ca289c76aa0-2", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_b0cdad17-e950-40cc-aad8-06117ef32f6d", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, hosts=>[//127.0.0.1], idle_flush_time=>1, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>>, @id="18be2952e4128a38734de3ec0ab20ca289c76aa0-2", @time_metric=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"18be2952e4128a38734de3ec0ab20ca289c76aa0-2", :events] key: duration_in_millis value: 0, @metric_events=#<LogStash::Instrument::NamespacedMetric:0xaef519c @metric=#<LogStash::Instrument::Metric:0x7c8cb53d @collector=#<LogStash::Instrument::Collector:0x2ab0d75d @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x28797b37 @store=#<Concurrent:0x00000000064380 entries=2 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x68bee688, @fast_lookup=#<Concurrent:0x00000000064384 entries=52 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"18be2952e4128a38734de3ec0ab20ca289c76aa0-2", :events]>, @output_class=LogStash::Outputs::ElasticSearch>", :error=>"SSLConnectionSocketFactory not found in packages org.apache.http.client.methods, org.apache.http.client.entity, org.apache.http.client.config, org.apache.http.config, org.apache.http.conn.socket, org.apache.http.impl, org.apache.http.impl.client, org.apache.http.impl.conn, org.apache.http.impl.auth, org.apache.http.entity, org.apache.http.message, org.apache.http.params, org.apache.http.protocol, org.apache.http.auth, java.util.concurrent, org.apache.http.client.protocol, org.apache.http.conn.ssl, java.security.cert, java.security.spec, java.security, org.apache.http.client.utils; last error: cannot load Java class org.apache.http.client.utils.SSLConnectionSocketFactory"}
[2017-12-20T14:25:08,278][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<NameError: SSLConnectionSocketFactory not found in packages org.apache.http.client.methods, org.apache.http.client.entity, org.apache.http.client.config, org.apache.http.config, org.apache.http.conn.socket, org.apache.http.impl, org.apache.http.impl.client, org.apache.http.impl.conn, org.apache.http.impl.auth, org.apache.http.entity, org.apache.http.message, org.apache.http.params, org.apache.http.protocol, org.apache.http.auth, java.util.concurrent, org.apache.http.client.protocol, org.apache.http.conn.ssl, java.security.cert, java.security.spec, java.security, org.apache.http.client.utils; last error: cannot load Java class org.apache.http.client.utils.SSLConnectionSocketFactory>, :backtrace=>["file:/C:/JavaDevelopment/logstash-5.6.0/vendor/jruby/lib/jruby.jar!/jruby/java/core_ext/module.rb:45:in const_missing'", "C:/JavaDevelopment/logstash-5.6.0/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:603:inssl_socket_factory_from_options'", "C:/JavaDevelopment/logstash-5.6.0/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:394:in pool_builder'", "C:/JavaDevelopment/logstash-5.6.0/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:402:inpool'", "C:/JavaDevelopment/logstash-5.6.0/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:208:in initialize'", "C:/JavaDevelopment/logstash-5.6.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:26:ininitialize'", "C:/JavaDevelopment/logstash-5.6.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:272:in `build_adapter'", "C:/JavaDevelopment/logstash-5.6.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch]-