Configure certificate for repository-s3 with ceph

[raphaelSeguin (Raphaël Seguin)]

Hi All,
I'm having a hard time setting up a repository with repository-s3 plugin and a ceph object store.
I'm using elasticsearch:7.13.2
I used the following configurations:
in elasticsearch.yml:

s3:
  client:
    default:
      endpoint: <ceph endpoint>
      proxy:
        host: <proxy host>
        port: <proxy port>

in systemd elasticsearch.service file, I'm adding access_key and secret key this way:

ExecStartPost=/bin/bash -c '/usr/bin/printf "{{ s3_access_key }}" | /usr/bin/docker exec -i {{ item.name }} bin/elasticsearch-keystore add --stdin s3.client.default.access_key'
ExecStartPost=/bin/bash -c '/usr/bin/printf "{{ s3_secret_key }}" | /usr/bin/docker exec -i {{ item.name }} bin/elasticsearch-keystore add --stdin s3.client.default.secret_key'

And then I'm adding the certificates like this (ceph_certificate.crt is a PEM formatted file)

xecStartPost=/bin/bash -c '/usr/bin/docker exec -i {{ item.name }} bin/elasticsearch-keystore add-file s3.client.default.picerts /etc/ssl/certs/ceph_certificate.crt'

After starting the docker containers on the three nodes i send a reload_secure_request which is successful ({"total":3,"successful":3,"failed":0})
Then I try to register a repository...

PUT /_snapshot/test2" 
{
  "type": "s3", 
  "settings": {
    "bucket": "my_ceph_container"
  }
}'

and i get this message:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[test2] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[test2] path  is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-cbbB0vfuSdeEby7-IRbtxw/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "sdk_client_exception: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
          "caused_by": {
            "type": "validator_exception",
            "reason": "validator_exception: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
            "caused_by": {
              "type": "sun_cert_path_builder_exception",
              "reason": "sun_cert_path_builder_exception: unable to find valid certification path to requested target"
            }
          }
        }
      }
    }
  },
  "status": 500
}

I guess I'm not putting the certificate in the right place. But I don't have any clue at the moment.
Thank you for your help.
(I already read SSLHandshakeException causes connections to fail | Elasticsearch Guide [master] | Elastic but it didn't tell me how to add the certificate to the keystore properly)

[DavidTurner]

@raphaelSeguin

elasticsearch-keystore add-file s3.client.default.picerts

That's not a setting that Elasticsearch knows about, so this won't do anything. Can you explain how you ended up trying to set a setting with this name? Maybe there's some bad docs somewhere that need fixing.

Repository connections use the JVM-wide trust store for certificate verification, so you can solve this by adding your own CA certificate to the JVM-wide truststore using keytool.

[raphaelSeguin (Raphaël Seguin)]

I can now stop messing around with elasticsearch-keystore.
I'll use jdk/bin/keytool in my container instead then. Would you know where this JVM-wide trust store sits in the elasticsearch container ? Or do I need to create this truststore from scratch and pass the java option in the ES_JAVA_OPTS environement variable like:

ES_JAVA_OPTS=-Xms2048m -Xmx2048m -Djavax.net.ssl.trustStore=/path/to/truststore.jks

Thanks !

[raphaelSeguin (Raphaël Seguin)]

I made a truststore

keytool -import -alias ceph -file ceph.crt -keystore ceph.jks

and then passed the file and password:

ES_JAVA_OPTS=-Xms2048m -Xmx2048m \
  -Djavax.net.ssl.trustStore=/usr/share/elasticsearch/config/ceph.jks \
  -Djavax.net.ssl.keyStorePassword=secretpasswordyoudontknow"

And now it works !
Thanks !!

[DavidTurner]

Great, thanks for working out the details and sharing them.

One thing to be aware of is that I think this will replace the JVM's trust store with your custom one containing a single trusted CA certificate. That will fix communications with your Ceph system but means that anything else in Elasticsearch that wants to validate a certificate with a different CA (e.g. a Watcher web hook) will likely now fail.

[DavidTurner]

I opened a feature request for support for more fine-grained control over the truststore used by a repository, rather than having to override the JVM-wide default:

Add support for per-repository truststore