Configuring SSL with Letsencrypt and sym links

Elastic Stack Elasticsearch
1

I am setting up and Elasticsearch server on a system that uses letsencrypt certificates.

The certificates and keys live under /etc/letsencrypt. There are several applications that need to access those key and rather than make copies of them I have created a new group called letsencrypt to which the users that need access to the keys belong.

I then created a sym link:

lrwxrwxrwx 1 root letsencrypt 52 Nov 10 13:43 /etc/elasticsearch/ssl -> /etc/letsencrypt/live/secmonprd08.its.auckland.ac.nz

on user Elasticsearch:

 sudo su - elasticsearch 
elasticsearch@secmonprd08:~$ ls -lL /etc/elasticsearch/ssl
total 24
-rw-r----- 1 root letsencrypt 2305 Nov  2 15:58 cert.pem
-rw-r----- 1 root letsencrypt 3750 Nov  2 15:58 chain.pem
-rw-r----- 1 root letsencrypt 6055 Nov  2 15:58 fullchain.pem
-rw-r----- 1 root letsencrypt 3272 Nov  2 15:58 privkey.pem
-rw-r--r-- 2 root root         682 Nov 16  2020 README

elasticsearch@secmonprd08:~$ head /etc/elasticsearch/ssl/cert.pem 
-----BEGIN CERTIFICATE-----
MIIGeTCCBWGgAwIBAgISBP7MuB1v+rytye62clRw/sVqMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTExMDIwMTU4NDBaFw0yMjAxMzEwMTU4MzlaMCkxJzAlBgNVBAMT
HnNlY21vbnByZDA4Lml0cy5hdWNrbGFuZC5hYy5uejCCAiIwDQYJKoZIhvcNAQEB
BQADggIPADCCAgoCggIBAKkvl55CY+aPr0Qi5t4nZgsj3WXIpQd0feO1JOsbNptN
lgdM3Rv/Q7W2MJmqB7dReieOQYwGswE9ge8F5o6stkvwnSkH3NzefxKut/ZTplD2
awxhhC7+p2PVzqH8yKYm81oOmRpYI0viaAcueE1mAwkC6R6ECJg7gFWbK7JmScHS
A3huIJBTFyVOMvKK2oiTfEd6mULqhv8XLOy0I0mJm7pY6eongEmoecFyMcyPu/Wu
mpJrgznQDfQndZGIE2lijN56oDkQsHiHPDnDXNB5T9pZNPIsVq8Y8SMq7n+YgVdp

ES appears to have full access to the files.

When I start Elasticsearch I get:

[2021-11-16T13:00:01,096][ERROR][o.e.x.c.s.SSLConfigurationReloader] [secmonprd08] failed to start watching directory [/etc/elasticsearch/ssl] for ssl configurations [[SSLConfiguration{keyConfig=[keyPath=[/etc/elasticsearch/ssl/privkey.pem], certPaths=[/etc/elasticsearch/ssl/fullchain.pem]], trustConfig=Combining Trust Config{JDK trusted certs, keyPath=[/etc/elasticsearch/ssl/privkey.pem], certPaths=[/etc/elasticsearch/ssl/fullchain.pem]}], cipherSuites=[[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.3, TLSv1.2, TLSv1.1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}, SSLConfiguration{keyConfig=[keyPath=[/etc/elasticsearch/ssl/privkey.pem], certPaths=[/etc/elasticsearch/ssl/fullchain.pem]], trustConfig=Combining Trust Config{JDK trusted certs, keyPath=[/etc/elasticsearch/ssl/privkey.pem], certPaths=[/etc/elasticsearch/ssl/fullchain.pem]}], cipherSuites=[[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.3, TLSv1.2, TLSv1.1]], sslClientAuth=[NONE], verificationMode=[FULL]}]] - java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/elasticsearch/ssl" "read")

It would appear that java can not read "/etc/Elasticsearch/ssl"

Any ideas what the problem is?

Russell

2

An AccessControlException is part of the Java Security Manager. It means that Elasticsearch is attempting to read a file outside of the configured directories (it's not a unix file permission problem)

What type of Elasticsearch install do you have (deb/rpm/tgz)?
Is your elasticsearch.yml in /etc/elasticsearch ?

3

Thanks Tim, I suspected as much. So I can't use symlinks which is a pain.

This means that I have to make copies to the keys in the in the actual config directory which is bad for security. I would much rather have one copy of the keys and tightly control access to the directory where the keys are stored.

4

I am using a directory symlink (e.g. /etc/Elasticsearch/certs/ -> /etc/ssl/certs/); that works.

It looks like your link also points to a directory.

1 Like
5

Symlinks should definitely work. I suspect something else is the problem here, which is where I was leading with my earlier questions.

Is /etc/elasticsearch really the configuration directory for your install (it might be, but I can't know for sure that it is due to the different deployment options).

6

yes, deployed from unbuntu package.

[2021-11-10T08:55:40,647][INFO ][o.e.n.Node               ] [secmonprd08] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms20g, -Xmx20g, -XX:+UseCompressedOops, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:MaxDirectMemorySize=10737418240, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=d
7

It does and that directory contains links to the actual cet/keys.

certbot generates new keys/certs with names that have and incrementing index in an archive directory and then changes the symlinks in the 'live' directory to point to the new certs.

I have linked the ssl dir to the live directory.

8

I have just tried reworking the links so now there is just one link in the chain but still getting the error from the java.

9

Perhaps your problem is 2 fold

Those letsencrypt files are

user:group
root:letsencrypt

And perm
640

So I suspect that elastsearch .. which runs as elasticsearch user simply can not read them because it does not belong to the same user or group nor has permission to read a other/ world

Perhaps try setting the perms on the letsencrypt files to 644

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.