Elasticsearch startup problemse

zx_emo · March 4, 2022, 9:49am

Hello, my Yml file looks like this

#xpack.license.self_generated.type: basic
xpack.security.enabled: true

#xpack.security.enrollment.enabled: true
#xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
#xpack.security.http.ssl.enabled: true
#keystore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/http.p12
xpack.security.transport.ssl.enabled: true
#
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  client_authentication: required
  keystore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/elastic-certificates.p12
  truststore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/elastic-certificates.p12
#
#xpack.security.transport.ssl.client_authentication: required
#
#xpack.security.transport.ssl.keystore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/elastic-certificates.p12
#
#xpack.security.transport.ssl.truststore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/elastic-certificates.p12
#
#cluster.initial_master_nodes: ["localhost.localdomain"]

discovery.type: single-node

When I started Elasticsearch with the above configuration I got an error with the following message

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:421) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) ~[?:?]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:143) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:131) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:265) [ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-8.0.1.jar:8.0.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) [elasticsearch-8.0.1.jar:8.0.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.1.jar:8.0.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:?]
        ... 33 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:?]
        ... 33 more

TimV · March 7, 2022, 12:09am

In the log files, just above the error traces, there should be a WARN message for DiagnosticTrustManager.
Can you paste that message here please?

zx_emo · March 7, 2022, 5:32am

@TimV

I'm not sure what I found was what you wanted

[WARN ][o.e.c.r.a. DiskThresholdMonitor] [localhost. localdomain] flood stage disk watermark [95%] exceeded on [-DzqsBQRTuG-bo8gVHgkDA][localhost.localdomain][/usr/software/es/elasticsearch-8.0.1/data] free: 1.3gb[2.7%], all indices on this node will be marked read-only 
[ERROR][o.e.i.g.GeoIpDownloader  ] [localhost.localdomain] exception during geoip databases update
java.net.SocketTimeoutException: Read timed out
	at sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283) ~[?:?]
	at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309) ~[?:?]
	at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) ~[?:?]
	at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) ~[?:?]
	at java.net.Socket$SocketInputStream.read(Socket.java:966) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:421) ~[?:?]
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580) ~[?:?]
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) ~[?:?]
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]
	at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
	at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:143) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:131) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:265) [ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-8.0.1.jar:8.0.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) [elasticsearch-8.0.1.jar:8.0.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.1.jar:8.0.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-03-07T00:24:57,620][WARN ][o.e.c.r.a.DiskThresholdMonitor] [localhost.localdomain] flood stage disk watermark [95%] exceeded on [-DzqsBQRTuG-bo8gVHgkDA][localhost.localdomain][/usr/software/es/elasticsearch-8.0.1/data] free: 1.3gb[2.7%], all indices on this node will be marked read-only
[2022-03-07T00:25:27,630][WARN ][o.e.c.r.a.DiskThresholdMonitor] [localhost.localdomain] flood stage disk watermark [95%] exceeded on [-DzqsBQRTuG-bo8gVHgkDA][localhost.localdomain][/usr/software/es/elasticsearch-8.0.1/data] free: 1.3gb[2.7%], all indices on this node will be marked read-only
[2022-03-07T00:25:32,250][INFO ][o.e.x.m.p.NativeController] [localhost.localdomain] Native controller process has stopped - no new native processes can be started

amitmbm · March 7, 2022, 5:59am

You don't have much disk space left on node running the Elasticsearch process, as mentioned in the below log.

[WARN ][o.e.c.r.a. DiskThresholdMonitor] [localhost. localdomain] flood stage disk watermark [95%] exceeded on [-DzqsBQRTuG-bo8gVHgkDA][localhost.localdomain][/usr/software/es/elasticsearch-8.0.1/data] free: 1.3gb[2.7%], all indices on this node will be marked read-only

Could you delete unused files to free up space, which is important as disk watermark reached flood state, it means you won't be able to write anything new in your Elasticsearch.

Also, your earlier errors were related to security configs, So were you able to fix it, if you want you can just enable the basic security to make sure you are able to up the service.

TimV · March 7, 2022, 6:38am

Please check the logs for the words DiagnosticTrustManager
There should be a message a few milliseconds before the SSLHandshakeException you posted.

zx_emo · March 7, 2022, 7:20am

I did not find the prompt you said whether it does not exist with elasticSearch.log

system · April 4, 2022, 7:21am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Xpack.security.transport.ssl setup issue Elasticsearch elastic-stack-security	4	372	October 5, 2022
Trying to figure out how to enable x-pack Elasticsearch elastic-stack-security	2	3147	January 28, 2020
Several question about SSL/TLS Elasticsearch elastic-stack-security	6	591	December 10, 2019
Starting ES with TLS options Elasticsearch elastic-stack-security	13	1287	October 14, 2019
Elasticsearch-setup-passwords ssl connection issue Elasticsearch elastic-stack-security	5	5047	February 12, 2019

Elasticsearch startup problemse

Related topics