I have update all my Elasticsearch cluster, and kibana to the version 8.8.0 from 8.5.3, when update Logstash it doesnt start runing and show the next error
[2023-06-08T13:06:33,163][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "cacert" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_certificate_authorities' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {......}
[2023-06-08T13:06:33,173][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "ssl_certificate_verification" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_verification_mode' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {......}
[2023-06-08T13:06:33,174][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "ssl" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_enabled' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {......}
[2023-06-08T13:06:33,915][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2023-06-08T13:06:33,944][ERROR][logstash.configmanagement.elasticsearchsource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2023-06-08T13:06:33,946][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.>, :backtrace=>["/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:67:in `with_license_check'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:43:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:41:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:94:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:363:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:287:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:90:in `<main>'"]}
[2023-06-08T13:06:33,953][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]
If i restore the old version it works fine.
Someone knows what happend? Thank you.
Hi Cristian,
cperzrt10:
[2023-06-08T13:06:33,163][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "cacert" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_certificate_authorities' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {......}
[2023-06-08T13:06:33,173][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "ssl_certificate_verification" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_verification_mode' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {......}
[2023-06-08T13:06:33,174][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "ssl" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_enabled' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {......}
Those are deprecation warnings that you can remediate using the recommendations in the particular warning.
cperzrt10:
[2023-06-08T13:06:33,915][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2023-06-08T13:06:33,944][ERROR][logstash.configmanagement.elasticsearchsource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2023-06-08T13:06:33,946][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.>, :backtrace=>["/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:67:in `with_license_check'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:43:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:41:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:94:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:363:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:287:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:90:in `<main>'"]}
If looks like Logstash is not able to connect to Elasticsearch. Can you check your connection and certificate information as per this thread ?
1 Like
If i dont update Logstash it work with ssl and certificates and all, the only thing that changes are the version, no one of the config files changes, then i udtade Logstash and it start trowing erros.
I've redone the ssl and certifiacte configuration and it doesn work
I've hit a known bug for Logstash in 8.8 when upgrade to 8.8.0/8.8.1 with xpack.management.elasticsearch.ssl.certificate_authority
configured in Logstash with centralized management setup.
The cause is a bug introduced after SSL standardization.
The solution is update the Elasticsearch output plugin to 11.15.8 or higher:
bin/logstash-plugin update logstash-output-elasticsearch
Double check version:
bin/logstash-plugin list --verbose | grep logstash-output-elasticsearch
OR
Specify ca_trusted_fingerprint setting in the logstash.yml
i.e. with CA ca.cert, it can be grabbed with
cat ca.cert | openssl x509 -outform der | sha256sum | awk '{print $1}'
Then set the following on logstash.yml using the output from the previous command (i.e. "abbc1bd27074502451217955ffcf203770659ca70735ec3a6e2a980c6ee73a5e
"):
xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: "abbc1bd27074502451217955ffcf203770659ca70735ec3a6e2a980c6ee73a5e"
xpack.management.elasticsearch.ssl.ca_trusted_fingerprint: "abbc1bd27074502451217955ffcf203770659ca70735ec3a6e2a980c6ee73a5e"
1 Like
Hello,
Not working for me after upgrading to 8.8.1.
(no plugin found on /bin/logstash also).
There is some link about this error:
opened 04:49PM - 25 May 23 UTC
closed 01:32PM - 05 Jun 23 UTC
bug
Team:Logstash
**Logstash information**:
Please include the following information:
1. Log… stash version (e.g. `bin/logstash --version`)
8.8.0
3. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
Docker
4. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
K8S
**Plugins installed**: (`bin/logstash-plugin list --verbose`)
Default
**JVM** (e.g. `java -version`):
Starting Logstash {"logstash.version"=>"8.8.0", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]"}
**OS version** (`uname -a` if on a Unix-like system):
**Description of the problem including expected versus actual behavior**:
After upgrading to 8.8.0 from 8.7.1 Logstash fails to start due to xpack management TLS certificate error message
**Steps to reproduce**:
Please include a *minimal* but *complete* recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
1. Upgrade from 8.7.1 to 8.8.0 with xpack.management.elasticsearch.ssl.certificate_authority configured
2.
3.
**Provide logs (if relevant)**:
```
logstash-1 logstash [2023-05-25T14:13:27,199][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
logstash-1 logstash [2023-05-25T14:13:27,201][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://REPLACED:9200/_xpack][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:url=>https://REPLACED:xxxxxx@REPLACED:9200/, :error_message=>"Elasticsearch Unreachable: [https://REPLACED:9200/_xpack][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash-1 logstash [2023-05-25T14:13:27,202][WARN ][logstash.licensechecker.licensereader] Attempt to validate Elasticsearch license failed. Sleeping for 0.04 {:fail_count=>2, :exception=>"Elasticsearch Unreachable: [https://REPLACED:9200/_xpack][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
logstash-1 logstash [2023-05-25T14:13:27,243][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
logstash-1 logstash [2023-05-25T14:13:27,254][ERROR][logstash.configmanagement.elasticsearchsource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
logstash-1 logstash [2023-05-25T14:13:27,255][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.>, :backtrace=>["/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:67:in `with_license_check'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:43:in `initialize'", "org/jruby/RubyClass.java:890:in `new'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:41:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:94:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:363:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:287:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:90:in `<main>'"]}
logstash-1 logstash [2023-05-25T14:13:27,259][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
```
config:
```
xpack.management.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/conf.d/ssl/elasticsearch-ca.pem"
xpack.management.elasticsearch.ssl.verification_mode: "certificate"
```
mounted secret in container:
```
logstash@logstash-0:~$ ls -lh /usr/share/logstash/conf.d/ssl/elasticsearch-ca.pem
lrwxrwxrwx 1 root logstash 27 May 13 22:23 /usr/share/logstash/conf.d/ssl/elasticsearch-ca.pem -> ..data/elasticsearch-ca.pem
```
opened 07:09AM - 07 Jun 23 UTC
question
I am experiencing an issue where I am unable to see logstash under Cluster overv… iew in my setup. Additionally, I am encountering the following error messages repeatedly:
![image](https://github.com/sherifabdlnaby/elastdocker/assets/24693946/a4478398-c15b-48de-9511-d783d39262ae)
```
elastic-logstash-1 | [2023-06-07T06:53:13,041][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
elastic-logstash-1 | [2023-06-07T06:53:13,706][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
elastic-logstash-1 | [2023-06-07T06:53:13,709][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
elastic-logstash-1 | [2023-06-07T06:53:43,041][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
elastic-logstash-1 | [2023-06-07T06:53:43,738][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
elastic-logstash-1 | [2023-06-07T06:53:43,744][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
elastic-logstash-1 | [2023-06-07T06:54:13,040][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
elastic-logstash-1 | [2023-06-07T06:54:13,771][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
elastic-logstash-1 | [2023-06-07T06:54:13,772][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
```
My configuration settings are set to the default values as follows:
*ELK 8.8.0
```
git clone https://github.com/sherifabdlnaby/elastdocker.git
make setup
make elk
```
I have checked the status of the logstash containers and they appear to be healthy:
```
[root@elk1 elastdocker]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fb3e1e604b3b elastdocker/logstash:8.8.0 "/usr/local/bin/dock…" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 0.0.0.0:9600->9600/tcp, :::9600->9600/tcp elastic-logstash-1
1aa62265a433 elastdocker/kibana:8.8.0 "/bin/tini -- /usr/l…" 6 minutes ago Up 6 minutes 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp elastic-kibana-1
ea7e23818a0a elastdocker/apm-server:8.8.0 "/usr/bin/tini -- /u…" 6 minutes ago Up 6 minutes 0.0.0.0:8200->8200/tcp, :::8200->8200/tcp elastic-apm-server-1
814c7980cd7b elastdocker/elasticsearch:8.8.0 "/bin/tini -- /usr/l…" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp elastic-elasticsearch-1
```
Anyone else have a similar issue too?
Case created on my side for an official recommandation
For the noob like me:
The real path is /usr/share/logstash, then apply the command provided below:
And the workaround is OK
1 Like
cperzrt10
(Cristian)
June 19, 2023, 10:03am
7
Yes the route /bin ... is relative the complete is what you've post
all is especified here
system
(system)
Closed
July 17, 2023, 10:04am
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.