Hello, I have installed elk setup 6.2.2 including x-pack, I have configured elasticsearch to use SSL since the problem started my logstash is not connecting to elasticsearch, it always trying to connect localhost:9200 even though I have given different IP, below is config file and logs
input {
beats {
client_inactivity_timeout => 1000
port => 5044
ssl => true
ssl_key => '/opt/elasticsearch/elkhost/elkhost.pkcs8.key'
ssl_certificate => '/opt/elasticsearch/elkhost/elkhost.crt'
ssl_verify_mode => 'none'
}
}
filter {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601} %{LOGLEVEL:loglevel}" ]
}
}
filter {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601} %{GREEDYDATA} (\[%{WORD:loglevel}\])" ]
}
}
filter {
if "beats_input_codec_plain_applied" in [tags] {
mutate {
remove_tag => ["beats_input_codec_plain_applied"]
}
}
}
filter {
if "_grokparsefailure" in [tags] {
mutate {
remove_tag => ["_grokparsefailure"]
}
}
}
output {
elasticsearch {
hosts => ["https://192.168.2.6:9200"]
cacert => /opt/elasticsearch/ca/ca.crt
user => elastic
password => password
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}
here is the logstash.yml file
xpack.monitoring.elasticsearch.url: "https://192.168.2.6:9200"
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "password"
xpack.monitoring.elasticsearch.ssl.ca: /opt/elasticsearch/ca/ca.crt
logstash service logs
[2018-06-14T07:23:53,895][WARN ][logstash.licensechecker.licensereader] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
No Available connections
[2018-06-14T07:23:53,895][ERROR][logstash.licensechecker.licensemanager] Unable to retrieve license information from license server {:message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError"}
[2018-06-14T07:23:53,895][WARN ][logstash.licensechecker.xpackinfo] Nil response from License Server
[2018-06-14T07:23:54,719][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>2, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2018-06-14T07:23:55,656][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-06-14T07:23:55,656][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-06-14T07:23:55,657][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-06-14T07:23:55,659][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-06-14T07:23:59,719][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>2, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2018-06-14T07:24:00,659][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-06-14T07:24:00,660][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-06-14T07:24:00,660][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-06-14T07:24:00,661][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-06-14T07:24:04,719][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>2, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2018-06-14T07:24:05,661][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-06-14T07:24:05,662][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-06-14T07:24:05,663][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}