Hi everybody,
I will start by saying that i am new to ELK, so i am trying to put up and ELK infrastructure with 2 Logstash servers, an elastic cluster with 2 nodes, and 1 kibana server.
I have succeed in securing the communication between the nodes in elastic cluster and also between elastic cluster and kibana via tls certificates generated with elastic-certutil.
But i have a problem with the connectivity between my logstash and my elastic cluster, as you see in my logs down bellow:
Logs from my logstash server:
"Mar 20 15:12:01 p-iot-logstash-01 systemd: Started logstash.
Mar 20 15:12:23 p-iot-logstash-01 systemd: Stopping logstash...
Mar 20 15:12:23 p-iot-logstash-01 systemd: logstash.service: main process exited, code=exited, status=143/n/a
Mar 20 15:12:23 p-iot-logstash-01 systemd: Stopped logstash.
Mar 20 15:12:23 p-iot-logstash-01 systemd: Unit logstash.service entered failed state.
Mar 20 15:12:23 p-iot-logstash-01 systemd: logstash.service failed.
[ERROR][logstash.javapipeline ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>, Mar 20 15:51:04 p-iot-logstash-01 systemd: Started logstash.
Mar 20 15:51:30 p-iot-logstash-01 logstash: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Mar 20 15:51:31 p-iot-logstash-01 logstash: [2020-03-20T15:51:31,777][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Mar 20 15:51:31 p-iot-logstash-01 logstash: [2020-03-20T15:51:31,954][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.1"}
Mar 20 15:51:34 p-iot-logstash-01 logstash: [2020-03-20T15:51:34,611][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Unrecognized SSL message, plaintext connection?"}
Mar 20 15:51:34 p-iot-logstash-01 logstash: [2020-03-20T15:51:34,677][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
Mar 20 15:51:36 p-iot-logstash-01 logstash: [2020-03-20T15:51:36,469][INFO ][org.reflections.Reflections] Reflections took 59 ms to scan 1 urls, producing 20 keys and 40 values
Mar 20 15:51:36 p-iot-logstash-01 logstash: [2020-03-20T15:51:36,857][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://logstash_system:xxxxxx@x.x.x.x:9200/, https://logstash_system:xxxxxx@x.x.x.x:9200/]}}
Mar 20 15:51:36 p-iot-logstash-01 logstash: [2020-03-20T15:51:36,959][ERROR][logstash.javapipeline ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:74:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:261:in health_check_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:270:in block in healthcheck!'", "org/jruby/RubyHash.java:1428:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:266:in healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:382:in update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:82:in update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:76:in start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:302:in build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:64:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:103:in create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:99:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch.rb:262:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.1-java/lib/logstash/outputs/elasticsearch/common.rb:27:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:106:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:200:in block in register_plugins'", "org/jruby/RubyArray.java:1814:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:199:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:501:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:212:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:154:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:109:in block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/cca.conf"], :thread=>"#<Thread:0x1514e867 run>"}
Mar 20 15:51:36 p-iot-logstash-01 logstash: [2020-03-20T15:51:36,997][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
Mar 20 15:51:37 p-iot-logstash-01 logstash: [2020-03-20T15:51:37,360][INFO ][logstash.agent
"
This is my config file for logstash:
"input {}
filter {}
output {
elasticsearch {
hosts => ["https://x.x.x.x:9200","https://x.x.x.x:9200"]
user => 'logstash_system'
password => 'mylogstashpassword'
manage_template => "false"
template_name => "logstash"
ssl => true
ssl_certificate_verification => true
cacert => "/etc/logstash/certs/ca.crt"
}
}
"
I have followed this guide during the installation process https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash
Thank you very much for your responses in advance