Elasticsearch startup problemse

Hello, my Yml file looks like this

#xpack.license.self_generated.type: basic
xpack.security.enabled: true

#xpack.security.enrollment.enabled: true
#xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
#xpack.security.http.ssl.enabled: true
#keystore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/http.p12
xpack.security.transport.ssl.enabled: true
#
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  client_authentication: required
  keystore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/elastic-certificates.p12
  truststore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/elastic-certificates.p12
#
#xpack.security.transport.ssl.client_authentication: required
#
#xpack.security.transport.ssl.keystore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/elastic-certificates.p12
#
#xpack.security.transport.ssl.truststore.path: /usr/software/es/elasticsearch-8.0.1/config/certs/elastic-certificates.p12
#
#cluster.initial_master_nodes: ["localhost.localdomain"]

discovery.type: single-node

When I started Elasticsearch with the above configuration I got an error with the following message

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:421) ~[?:?]
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580) ~[?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) ~[?:?]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:143) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:131) ~[ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:265) [ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-8.0.1.jar:8.0.1]
        at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-8.0.1.jar:8.0.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) [elasticsearch-8.0.1.jar:8.0.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.1.jar:8.0.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:?]
        ... 33 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:?]
        ... 33 more

In the log files, just above the error traces, there should be a WARN message for DiagnosticTrustManager.
Can you paste that message here please?

@TimV

I'm not sure what I found was what you wanted

[WARN ][o.e.c.r.a. DiskThresholdMonitor] [localhost. localdomain] flood stage disk watermark [95%] exceeded on [-DzqsBQRTuG-bo8gVHgkDA][localhost.localdomain][/usr/software/es/elasticsearch-8.0.1/data] free: 1.3gb[2.7%], all indices on this node will be marked read-only 
[ERROR][o.e.i.g.GeoIpDownloader  ] [localhost.localdomain] exception during geoip databases update
java.net.SocketTimeoutException: Read timed out
	at sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283) ~[?:?]
	at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309) ~[?:?]
	at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) ~[?:?]
	at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) ~[?:?]
	at java.net.Socket$SocketInputStream.read(Socket.java:966) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:421) ~[?:?]
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580) ~[?:?]
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) ~[?:?]
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]
	at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
	at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:143) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:131) ~[ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:265) [ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-8.0.1.jar:8.0.1]
	at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-8.0.1.jar:8.0.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) [elasticsearch-8.0.1.jar:8.0.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.1.jar:8.0.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-03-07T00:24:57,620][WARN ][o.e.c.r.a.DiskThresholdMonitor] [localhost.localdomain] flood stage disk watermark [95%] exceeded on [-DzqsBQRTuG-bo8gVHgkDA][localhost.localdomain][/usr/software/es/elasticsearch-8.0.1/data] free: 1.3gb[2.7%], all indices on this node will be marked read-only
[2022-03-07T00:25:27,630][WARN ][o.e.c.r.a.DiskThresholdMonitor] [localhost.localdomain] flood stage disk watermark [95%] exceeded on [-DzqsBQRTuG-bo8gVHgkDA][localhost.localdomain][/usr/software/es/elasticsearch-8.0.1/data] free: 1.3gb[2.7%], all indices on this node will be marked read-only
[2022-03-07T00:25:32,250][INFO ][o.e.x.m.p.NativeController] [localhost.localdomain] Native controller process has stopped - no new native processes can be started

You don't have much disk space left on node running the Elasticsearch process, as mentioned in the below log.

[WARN ][o.e.c.r.a. DiskThresholdMonitor] [localhost. localdomain] flood stage disk watermark [95%] exceeded on [-DzqsBQRTuG-bo8gVHgkDA][localhost.localdomain][/usr/software/es/elasticsearch-8.0.1/data] free: 1.3gb[2.7%], all indices on this node will be marked read-only

Could you delete unused files to free up space, which is important as disk watermark reached flood state, it means you won't be able to write anything new in your Elasticsearch.

Also, your earlier errors were related to security configs, So were you able to fix it, if you want you can just enable the basic security to make sure you are able to up the service.

Please check the logs for the words DiagnosticTrustManager
There should be a message a few milliseconds before the SSLHandshakeException you posted.

I did not find the prompt you said whether it does not exist with Elasticsearch.log

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.