S3 Repo plugin failing certificate validation with MiniO, already added cert to Jdk truststore

[R.L]

I want to take snapshots of my ElasticSearch Index and use MiniO instance running on a remote machine as storage. My ES instance (7.6.0) is ran locally, using JDK 13.

I can access the miniO storage through my browser just fine.

I have already added the certs used by MiniO into my Jdk truststore like so,

keytool -import -trustcacerts -storepass xxx -alias test_minio_trust -file rootCA.crt -keystore truststore_root_minio.jks

However, when I try to create repo:

    {
      "type": "s3",
      "settings": {
        "bucket": "esrepo",
        "base_path": "",
        "endpoint": "<remote_ip>:9000/",
        "protocol": "https"	
      }
    }

I get:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_backup] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_backup] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-lPkWVIy5RqmHems-7KvT9w/master.d
at] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Unable to execute HTTP request: PKIX path building failed: s
un.security.provider.certpath.SunCertPathBuilderException: unable to find valid
certification path to requested target",
        "caused_by" : {
          "type" : "s_s_l_handshake_exception",
          "reason" : "PKIX path building failed: sun.security.provider.certpath.
SunCertPathBuilderException: unable to find valid certification path to requeste
d target",
          "caused_by" : {
            "type" : "validator_exception",
            "reason" : "PKIX path building failed: sun.security.provider.certpat
h.SunCertPathBuilderException: unable to find valid certification path to reques
ted target",
            "caused_by" : {
              "type" : "sun_cert_path_builder_exception",
              "reason" : "unable to find valid certification path to requested t
arget"
            }
          }
        }
      }
    }
  },
  "status" : 500
}

Which seems like a cert validation error to me. I am using a self-signed cert but my impression is that adding the cert to JDK is suppose to eliminate validation errors. What am I doing wrong? Is ES not picking up the cert?

[Andre_Letterer (André Letterer)]

Hi,

welcome to the Elastic Community.

Yes, exactly, the main error is this one:
unable to find valid certification path to requested target

First of all, the S3 Plugin itself is a basically more or less a hat over the AWS SDK.
Having that said, it would not make sense to export the root certificate to an own JKS store like in your case truststore_root_minio.jks . You should use the normal default cacerts keystore that the JVM uses normally under lib/security of the JDK home directory.

For adding the root CA certificate to this keystore you should either convert your certificate from PEM to DER format before using keytool, or you would use keystore explorer to import the certificate to the cacerts file.
Remember that the same cacerts needs to be known to every elasticsearch node.

And it will need a restart of the elasticsearch node.

After words the snapshots should work.

Let me know how it goes.

[R.L]

Thank you for your reply, I talked it over with stakeholder and they mentioned they didn't need cert validation for S3-like connections as we are only going to need them for MiniO being ran within our environment. We will not need to run connection against anything external. so I just ended up disabling cert validation via JVM option:

export ES_JAVA_OPTS="$ES_JAVA_OPTS -Dcom.amazonaws.sdk.disableCertChecking=true"

Thanks for the help though, at least now I know how I -would- get it to work.

[Andre_Letterer (André Letterer)]

Ah, ok. Good to know that there is an option like that. But in my opinion, instead of ignoring Certificate Checks the better choice are always to keep certificate checks at least for production use cases.
However if you talked through the risks, then it's completely fine.