Can’t start logstash after x-pack installation

Elastic Stack Logstash
1

I have this issue now as well, I had logstash running and connecting without issue, after installing x-pack it is now looking at localhost for elastic which doesn't exist.
Elastic 6.1.1, logstash 6.1.1 Windows 2012R2.
Config,
input {
beats { port => 5044
}
}
output {
elasticsearch {
hosts => ["elastic1:9200"]
}
stdout { codec => rubydebug }

output:
[2018-01-12T19:18:35,705][INFO ][logstash.licensechecker.licensereader] Running
health check to see if an Elasticsearch connection is working {:healthcheck_url=

http://localhost:9200/, :path=>"/"}

2

I have this in the logstash.yml file,
xpack.license.self_generated.type: basic
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: "http://elastic1:9200"

3

Did you install X-Pack in Elasticsearch as well?

4

Yes, x-pack is installed on all nodes and kibana

5

I don't think you need that.

Also, what do the rest of the logs show?

6

Here you go,

[2018-01-12T19:17:58,972][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"D:/logstash/modules/fb_apache/configuration"}
[2018-01-12T19:17:58,988][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"D:/logstash/modules/netflow/configuration"}
[2018-01-12T19:18:00,409][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"D:/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.1.1-java/modules/arcsight/configuration"}
[2018-01-12T19:18:00,722][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-01-12T19:18:01,769][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.1.1"}
[2018-01-12T19:18:02,284][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-01-12T19:18:05,128][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch hosts=>[http://localhost:9200], bulk_path=>"/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s", manage_template=>false, document_type=>"%{[@metadata][document_type]}", sniffing=>false, id=>"ad524e5a1a68d2ca7086e1144ec98005bcfc1ad3103a990fb9bbf21aa44aa140", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_4913a6bd-22a9-48f8-b3f8-0c2db2c1a1b7", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2018-01-12T19:18:05,769][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-01-12T19:18:05,784][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-01-12T19:18:08,034][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused: connect"}
[2018-01-12T19:18:08,050][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2018-01-12T19:18:08,065][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2, :thread=>"#<Thread:0x39ca0750 run>"}
[2018-01-12T19:18:08,253][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-01-12T19:18:08,253][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-01-12T19:18:10,315][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused: connect"}
[2018-01-12T19:18:12,378][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused: connect {:url=>http://localhost:9200/, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused: connect", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-01-12T19:18:12,393][ERROR][logstash.licensechecker.licensemanager] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused: connect", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-01-12T19:18:12,393][WARN ][logstash.licensechecker.xpackinfo] Nil response from License Server
[2018-01-12T19:18:12,456][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2018-01-12T19:18:13,128][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-01-12T19:18:15,206][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused: connect"}
[2018-01-12T19:18:15,221][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic1:9200/]}}
[2018-01-12T19:18:15,237][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic1:9200/, :path=>"/"}
[2018-01-12T19:18:15,284][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic1:9200/"}
[2018-01-12T19:18:15,346][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>nil}
[2018-01-12T19:18:15,346][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-01-12T19:18:15,362][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}

7

Looks ok to me?

1 Like
8

I see it is only a warning, but it is spamming the log window and I was hoping the Logstash would show up on the kibana but it isn't .

9

More info?
The below error is spamming my log every two minutes and I see a connection to elastic but I cant see any data getting to it. Any ideas? do you need more info?
I was thinking of installing elastic on the logstash as an ingest node, but I don't know if that would help?

[WARN ][logstash.licensechecker.licensereader] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2018-01-17T18:47:32,814][ERROR][logstash.licensechecker.licensemanager] Unable to retrieve license information from license server {:message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError"}

10

can you confirm that your logstash.yml file doesn't have lines that start with empty spaces?
It seems that logstash isn't processing those settings correctly so it's defaulting to connect to localhost

Something like:

 xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: "http://elastic1:9200"

Or similar could be causing this issue.

11

Made some progress, cleaned up the yml file and removed the x-pack ssl settings, that said now when I try to connect I am getting this,
[logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance,
but got an error. {:url=>"https://elastic1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic1:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

So I think that means it isn't reading the ca cert maybe?
I have this in the conf file, am I missing anything?

input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => ["D:\logstash\config\ca\ca.crt"]
ssl_certificate => "D:\logstash\config\logstash\logstash.crt"
ssl_key => "D:\logstash\config\logstash\logstash.key"

}

}
output {
elasticsearch {
hosts => ["https://elastic1:9200"]
ssl => true
cacert => "D:/logstash/config/ca/ca.crt"

}

stdout { codec => rubydebug }
}

12

Please show your logstash.yml file.

13

I only included uncommented settings.

node.name: logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: "https://elastic1:9200"
xpack.monitoring.elasticsearch.sniffing: true
dead_letter_queue.enable: true
dead_letter_queue.max_bytes: 1024mb
http.host: "10.210.0.161"

14

I've tried reversing the slash's in the config file, that didn't help.

15

Found a solution, Import the CA into the Java keystore, note do not use IP's in the cert, Logstash will spam errors about the ip address.

"C:\Program Files\Java\jre1.8.0_151\bin\keytool" -import -alias ca -keystore "C:\Program Files\Java\jre1.8.0_151\lib\security\cacerts" -file "D:\logstash\config\ca\ca.crt"

16

[2018-02-05T23:20:02,277][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-02-05T23:20:02,280][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

I'm facing a similar problem. I don't understand why localhost:9200 is mentioned in the logs when I've given the IP address of elasticsearch in both the config file and logstash.yml file.

Any help will be appreciated. Thanks!

17

Please start a new thread with your question :slight_smile:

18

Sure thanks. Please check LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/]

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.