Help needed for certificate configuration

Elasticsearch drives me nuts when it comes to certificates and how they are used / configured.

This is my current configuration:

## Cluster Settings
cluster.name: "elk-tls-cluster"
node.name: node-1
network.host: "0.0.0.0"
http.host: 0.0.0.0

## License
xpack.license.self_generated.type: basic

# Security
xpack.security.enabled: true
xpack.security.authc.token.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.transport.ssl.verification_mode: certificate

# transport security settings
# This is mostly used for inter-node communications between parts of the ELK stack
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: ${CONFIG_DIR}/elasticsearch.p12
xpack.security.transport.ssl.truststore.path: ${CONFIG_DIR}/elasticsearch_ca.p12


# HTTP security setttings
# This is used for client server ssl/tls communications (e.g. browser to kibana)
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: ${CONFIG_DIR}/elasticsearch.p12
xpack.security.http.ssl.truststore.path: ${CONFIG_DIR}/elasticsearch_ca.p12

"elasticsearch.p12" is the keystore containing my http certificates:

[elasticsearch]# keytool -list -v -keystore elasticsearch.p12
Enter keystore password:

*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: http
Creation date: Dec 12, 2023
Entry type: PrivateKeyEntry


*******************************************
*******************************************

"elasticsearch_ca.p12" is a keystore where I imported the root certificate (selfsigned root CA):

[elasticsearch]# keytool -list -v -keystore elasticsearch_ca.p12
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: root
Creation date: Dec 12, 2023
Entry type: trustedCertEntry

Owner: CN=Elastic Certificate Tool Autogenerated CA
Issuer: CN=Elastic Certificate Tool Autogenerated CA
Serial number: a2aadf5f4b5b675076f66712155e3614aeb82191
Valid from: Tue Feb 14 15:18:04 CET 2023 until: Fri Feb 13 15:18:04 CET 2026
Certificate fingerprints:
    MD5:  87:8D:E0:F7:F3:38:BB:EF:07:6B:5C:60:3C:D2:F9:07
    SHA1: 88:FC:2D:F9:19:3E:57:1F:74:73:5D:E7:01:11:7C:14:B5:BD:6C:98
    SHA256: 02:8B:E9:90:25:82:AD:8B:47:60:21:FB:23:6B:C8:5E:72:DE:B5:DD:86:85:3A:7A:C9:3E:89:44:09:E1:29:57
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BA 29 BD F2 F4 7A E2 1A   28 C4 F8 6B 18 FD D8 83  .)...z..(..k....
0010: F0 31 73 7C                                        .1s.
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: BA 29 BD F2 F4 7A E2 1A   28 C4 F8 6B 18 FD D8 83  .)...z..(..k....
0010: F0 31 73 7C                                        .1s.
]
]



*******************************************
*******************************************

My problem is that I cannot use elasticsearch command line tools like "elasticsearch-reset-password" or "elasticsearch-create-enrollment-token"

I always the the following error message:

20:26:57.289 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [elasticsearch]; the server provided a certificate with subject name [CN=elasticsearch], fingerprint [fc50d2fa054952318e2b57c601001a1b6f81d32b], no keyUsage and no extendedKeyUsage; the certificate is valid between [2023-02-14T14:18:07Z] and [2026-02-13T14:18:07Z] (current time is [2023-12-12T20:26:57.286086601Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:elasticsearch.fire.fly,DNS:localhost,IP:127.0.0.1,IP:0.0.0.0,DNS:elasticsearch]; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [52e7f3cffa59d6e4dc8f0a47c1d1c08560265ede]) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is not trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: StoreTrustConfig{path=/usr/share/elasticsearch/config/elasticsearch_ca.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX})]); this ssl context does trust a certificate with subject [CN=Elastic Certificate Tool Autogenerated CA] but the trusted certificate has fingerprint [88fc2df9193e571f74735de701117c14b5bd6c98]

I guess the truststore needs to be configured on some other locations - but wondering where.
Any ideas?

you need to add the -url parameter and use the url that matches the cert

on of these

[DNS:elasticsearch.fire.fly,DNS:localhost,IP:127.0.0.1,IP:0.0.0.0,DNS:elasticsearch]

I know - the error message is different missing that parameter saying thet there is an issue with the CN of the certificate :slight_smile:

The full command I am using is as follows:

elasticsearch-reset-password -u 'username' -i --url https://elasticsearch:9200

or when it comes to token creation:

elasticsearch-create-enrollment-token -s kibana --url https://elasticsearch:9200

In both cases I get the error message above.

HI @litronics

This is why it is important/good practice to share the entire / complete command in the first place... and the results.. otherwise we are operating on incomplete information.

The error messages you have above are usually created when you're running the command line without the URL parameter which then just uses the IP address or local host which doesn't match common name or subject alternative names in the cert.

So now that I understand you're using that parameter leads to another set of questions.

Exactly How did you create the HTTP SSL certs?

Also could you try another URL you have in the SAN like 127.0.0.1

I do not think this is valid.

Exactly How did you create the HTTP SSL certs?

I used a docker container and a script to create the required certificates and keystores required for all my instances. That works actually pretty well and automatically creates a CA certificate and signs all SSL certificates with this CA.

You are actually right - the IP of 0.0.0.0 is not necessarily valid. However, the SAN entries are checked if the CN is not matching. If there is also no match in the SAN entries the error message says that there are an issue with the CN or the certificate is not valid for this URL / service. So, this is not the root of my issues.

After digging around and looking deeper in the keystore I found that there are three certificates included (http, elasticsearch, root) where the root certificate is self signed (as normal).

So my current understanding is that the aliases http and elasticsearch are used by the xpack.security.http.ssl respectively by xpack.security.transport.ssl settings.
The root alias was previously used by the truststore settings (for both transport and http) as all settings were pointing to a single keystore.

Due to security changes the truststore has to be separated from the keystore and therefore I added the CA certificate to a truststore and referenced this new keystore in the configuration leaving the root certificate within the older keystore.

Actually, removing the root alias from the keystore enables the elasticsearch-reset-password command. But brings up a totally new error message:

elasticsearch@elasticsearch:~$ elasticsearch-create-enrollment-token -s kibana --url https://elasticsearch:9200
Unable to create enrollment token for scope [kibana]

ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration Keystore doesn't contain any PrivateKey entries where the associated certificate is a CA certificate

So I verified that the keystore hast the private keys included for both aliases.

elasticsearch@elasticsearch:~$ jdk/bin/keytool -list -v -keystore config/elasticsearch.p12
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

Alias name: elasticsearch
Creation date: Dec 13, 2023
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=elasticsearch
Issuer: CN=Elastic Certificate Tool Autogenerated CA
Serial number: f18f18af5c22c9417674415217dc80100e029d48
Valid from: Tue Feb 14 14:18:09 GMT 2023 until: Fri Feb 13 14:18:09 GMT 2026
Certificate fingerprints:
         SHA1: FA:A5:11:D0:B0:D7:60:20:79:D5:ED:5B:1E:E6:CB:08:67:A5:43:72
         SHA256: 1B:3D:2F:B5:A3:BF:1D:D7:CC:1C:9E:95:92:95:2F:77:BE:54:54:BC:12:FF:77:78:41:64:60:1B:D0:72:23:07
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BA 29 BD F2 F4 7A E2 1A   28 C4 F8 6B 18 FD D8 83  .)...z..(..k....
0010: F0 31 73 7C                                        .1s.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

#3: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: elasticsearch.fire.fly
  DNSName: localhost
  IPAddress: 127.0.0.1
  IPAddress: 0.0.0.0
  DNSName: elasticsearch
]

#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 5A 5E E9 3C 4C F0 FF BE   3E 6A 43 2B DB FE E1 73  Z^.<L...>jC+...s
0010: B6 E5 98 E9                                        ....
]
]



*******************************************
*******************************************


Alias name: http
Creation date: Dec 13, 2023
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=elasticsearch
Issuer: CN=Elastic Certificate Tool Autogenerated CA
Serial number: f18f18af5c22c9417674415217dc80100e029d48
Valid from: Tue Feb 14 14:18:09 GMT 2023 until: Fri Feb 13 14:18:09 GMT 2026
Certificate fingerprints:
         SHA1: FA:A5:11:D0:B0:D7:60:20:79:D5:ED:5B:1E:E6:CB:08:67:A5:43:72
         SHA256: 1B:3D:2F:B5:A3:BF:1D:D7:CC:1C:9E:95:92:95:2F:77:BE:54:54:BC:12:FF:77:78:41:64:60:1B:D0:72:23:07
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BA 29 BD F2 F4 7A E2 1A   28 C4 F8 6B 18 FD D8 83  .)...z..(..k....
0010: F0 31 73 7C                                        .1s.
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

#3: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: elasticsearch.fire.fly
  DNSName: localhost
  IPAddress: 127.0.0.1
  IPAddress: 0.0.0.0
  DNSName: elasticsearch
]

#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 5A 5E E9 3C 4C F0 FF BE   3E 6A 43 2B DB FE E1 73  Z^.<L...>jC+...s
0010: B6 E5 98 E9                                        ....
]
]



*******************************************
*******************************************

I also verified that the truststore doesn't contain der CAs private key:

elasticsearch@elasticsearch:~$ jdk/bin/keytool -list -v -keystore config/elasticsearch_ca.p12
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: root
Creation date: Dec 12, 2023
Entry type: trustedCertEntry

Owner: CN=Elastic Certificate Tool Autogenerated CA
Issuer: CN=Elastic Certificate Tool Autogenerated CA
Serial number: a2aadf5f4b5b675076f66712155e3614aeb82191
Valid from: Tue Feb 14 14:18:04 GMT 2023 until: Fri Feb 13 14:18:04 GMT 2026
Certificate fingerprints:
         SHA1: 88:FC:2D:F9:19:3E:57:1F:74:73:5D:E7:01:11:7C:14:B5:BD:6C:98
         SHA256: 02:8B:E9:90:25:82:AD:8B:47:60:21:FB:23:6B:C8:5E:72:DE:B5:DD:86:85:3A:7A:C9:3E:89:44:09:E1:29:57
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BA 29 BD F2 F4 7A E2 1A   28 C4 F8 6B 18 FD D8 83  .)...z..(..k....
0010: F0 31 73 7C                                        .1s.
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen: no limit
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: BA 29 BD F2 F4 7A E2 1A   28 C4 F8 6B 18 FD D8 83  .)...z..(..k....
0010: F0 31 73 7C                                        .1s.
]
]



*******************************************
*******************************************

Looking deeper at the error message

Elasticsearch node HTTP layer SSL configuration Keystore doesn't contain any PrivateKey entries where the associated certificate is a CA certificate.

I am assuming wir are talking about "xpack.security.http.ssl" part of the settings and the http alias is containing it's PrivateKey. Wondering what PrivateKey it means or if I need to include the root certificate in the keystore. But later would then break the "xpack.security.transprot.ssl" setting again.

I am also wondering what the certificate best practices are.
Do I need three different keystores based on the settings area (HTTP, TRANSPORT and TRUSTSTORE)?

No just http and transport.

It seems like you are combining them... that is usually not considered best practice but should work.

But I think I saw something...

Perhaps you should review... the normal setup...
Transport Security and HTTP Security

Also you can see the default setup by just running

docker run --name es01 --net elastic -p 9200:9200 -it -m 1GB docker.elastic.co/elasticsearch/elasticsearch:8.11.1
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

Thank you so much for your support Stephen!

Here is what I tried:

  1. Upgrade to the latest version - running

Version: 8.11.1, Build: docker/6f9ff581fbcde658e6f69d6ce03050f060d1fd0c/2023-11-11T10:05:59.421038163Z, JVM: 21.0.1

  1. recreated the http.p12 keystore with elasticsearch-certutil http

This didn't change the behavior :frowning:

  1. disabled the security configuration as per the following configuration:
## Cluster Settings
cluster.name: "elk-tls-cluster"
node.name: node-1

## License
xpack.license.self_generated.type: basic

# Security
xpack.security:
  enabled: false
  authc.api_key.enabled: true
  enrollment.enabled: true

# transport security settings
# This is mostly used for inter-node communications between parts of the ELK stack
#
xpack.security.transport.ssl:
  enabled: false
  verification_mode: "none"
# crt/key configuration
  #key: ${CONFIG_DIR}/elasticsearch.key
  #certificate: ${CONFIG_DIR}/elasticsearch.crt
  #certificate_authorities: ${CONFIG_DIR}/ca.crt

# Keystore configuration
  keystore.path: ${CONFIG_DIR}/elasticsearch.p12
  truststore.path: ${CONFIG_DIR}/elasticsearch_ca.p12


# HTTP security setttings
# This is used for client server ssl/tls communications (e.g. browser to kibana)
xpack.security.http.ssl:
  enabled: false
# crt/key configuration
  #key: ${CONFIG_DIR}/elasticsearch.key
  #certificate: ${CONFIG_DIR}/elasticsearch.crt
  #certificate_authorities: ${CONFIG_DIR}/ca.crt

# Keystore configuration
  keystore.path: ${CONFIG_DIR}/elasticsearch.p12
  truststore.path: ${CONFIG_DIR}/elasticsearch_ca.p12

# Repo Path for Snapshots
path:
  repo:
    - /usr/share/elasticsearch/backups
  1. needed to delete all DATA as some shards didn't came up and the cluster status stuck at yellow. Cluster status is now:
elasticsearch@elasticsearch:~$ curl localhost:9200/_cat/health
1702539754 07:42:34 elk-tls-cluster green 1 1 0 0 0 0 0 0 - 100.0%
  1. tried to create a enrollment token for kibana:
elasticsearch@elasticsearch:~$ elasticsearch-create-enrollment-token -s kibana --url http://localhost:9200
Unable to create enrollment token for scope [kibana]

ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration Keystore doesn't contain any PrivateKey entries where the associated certificate is a CA certificate, with exit code 73

Have I missed something ??

tried to strip down the instance further more and removed all unnecessary settings from the elasticsearch.yml

Leaving it like this:

## Cluster Settings
cluster.name: "elk-tls-cluster"
node.name: node-1

## License
xpack.license.self_generated.type: basic

# Repo Path for Snapshots
path:
  repo:
    - /usr/share/elasticsearch/backups

I learned that I also need to remove the keystore passwords from the elasticsearch-keystore - otherwise ES is not starting.

Next learning was, that I have to add xpack.security.enrollment.enabled to my config:

## Cluster Settings
cluster.name: "elk-tls-cluster"
node.name: node-1

## License
xpack.license.self_generated.type: basic

## Security
xpack.security.enrollment.enabled: true

## Repo Path for Snapshots
path:
  repo:
    - /usr/share/elasticsearch/backups

Finally I am totally confused as I am still not able to create a enrollment token:

elasticsearch@elasticsearch:~$ elasticsearch-create-enrollment-token -s kibana --url http://localhost:9200
Unable to create enrollment token for scope [kibana]

ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration is not configured with a keystore, with exit code 73

Hi @litronics, I am sorry this is turning out to be an issue.

1st, let's back up and try to understand exactly what you are trying to accomplish.

Are you trying to setup

  1. A SIngle Node Cluster with Kibana with Security?
  2. A multimode Node Cluster with Security?
  3. Are you trying to manually configure security or use Auto Configuration?
  4. What is your endstate?
  5. What OS and install method did you use, tar.gz, docker, rpm?
    ....

Looking at your last post, I ask because I think you are making some incorrect assumptions.

1st - Let me clear up something top-level. You have 2 choices when setting up security.

  1. Setp with security enabled automatically
  2. Manually Setup Security

You are mixing the 2 you can not manually or partially setup Security and then use the Enrollment Token method that will not work. See details here

So tell me what you are trying to accomplish and then lets back up a bit..

If this has been about running the enrollment token from the get-go... after you manually configured security, that explains a lot. My apologies; I should have caught that sooner!!!

I was focusing in the reset-password.....

Thanks Stephen,

I guess your are right - somewhere I walked in the wrong direction while troubleshooting these error messages.

Therefore, I started from scratch - created new selfsigned CA and all required certificates with
bin/elasticsearch-certutil cert --silent --in $CONFIG_DIR/instances.yml --out $CERT_KEYSTORES_ZIP --ca $CA_P12 --ca-pass "" --pass "" . Basically I am running a script in a elasticsearch container creating a secrets folder with all certs required.

I also deleted all volumes and moved the configuration form the yml-files into environmen variables in my docker-compose.yml.

Then I started the containers in the following sequence:

  1. elasticsearch
  2. kibana
  3. fleet-server

I successfully configured elasticsearch and kibana.
But with fleet-server I do have some issues with the CA certificate:

Here is the used docker-compose file (the important part of it):

version: '3.1'

services:
  proxy:
    image: jwilder/nginx-proxy
    container_name: elk_proxy
    ports:
       - "443:443"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./proxy/certs:/etc/nginx/certs:ro
      - ./proxy/nginx/vhost.d:/etc/nginx/vhost.d
      - ./proxy/nginx/html:/usr/share/nginx/html
      - ./proxy/nginx/proxy.conf:/etc/nginx/proxy.conf:ro
    depends_on:
      - elasticsearch
      - kibana
    labels:
      - "com.github.jrcs.letsencrypt_nginx_proxy_companion.proxy"
    networks:
      dmz:
          ipv4_address: 192.168.25.20
      elk:

  elasticsearch:
    container_name: elk_elasticsearch
    hostname: elasticsearch
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: ${ELK_VERSION}
    deploy:
      resources:
        limits:
          cpus: "1"
          memory: "20GB"
    restart: always
    environment:
      CONFIG_DIR: ${ELASTIC_DIR}/config
      node.name: es01
      discovery.seed_hosts: es01
      discovery.type: single-node
      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
      ES_JAVA_OPTS: -Xmx${ELASTICSEARCH_HEAP} -Xms${ELASTICSEARCH_HEAP}
      bootstrap.memory_lock: "true"
      xpack.license.self_generated.type: basic
      xpack.security.enabled: true
      xpack.security.http.ssl.enabled: true
      xpack.security.http.ssl.key: elasticsearch.key
      xpack.security.http.ssl.certificate_authorities: ca.crt
      xpack.security.http.ssl.certificate: elasticsearch.crt
      xpack.security.transport.ssl.enabled: true
      xpack.security.transport.ssl.verification_mode: certificate
      xpack.security.transport.ssl.certificate_authorities: ca.crt
      xpack.security.transport.ssl.certificate: elasticsearch.crt
      xpack.security.transport.ssl.key: elasticsearch.key
      VIRTUAL_HOST: elasticsearch.fire.fly
      VIRTUAL_PORT: 9200
      VIRTUAL_PROTO: https
      LETSENCRYPT_HOST: elasticsearch.fire.fly
    volumes:
      - ES_data:${ELASTIC_DIR}/data
      - backup:${ELASTIC_DIR}/backup
    secrets:
      - source: ca.crt
        target: ${ELASTIC_DIR}/config/ca.crt
      - source: elasticsearch.cert
        target: ${ELASTIC_DIR}/config/elasticsearch.crt
      - source: elasticsearch.key
        target: ${ELASTIC_DIR}/config/elasticsearch.key
    ports:
      - "9200:9200"
      - "9300:9300"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 200000
        hard: 200000
    healthcheck:
      test: curl --cacert config/ca.crt -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5
    networks:
      - elk

  kibana:
    container_name: elk_kibana
    hostname: kibana
    build:
      context: kibana/
      args:
        ELK_VERSION: ${ELK_VERSION}
    restart: always
    environment:
      CONFIG_DIR: ${KIBANA_DIR}/config
      ENCRYPTION_KEY: ${XPACK_ENCRYPTION_KEY}
      ELASTICSEARCH_HOSTS: ${ELASTIC_URL}
      ELASTICSEARCH_USERNAME: ${KIBANA_USERNAME}
      ELASTICSEARCH_PASSWORD: ${KIBANA_PASSWORD}
      KIBANA_URL: ${KIBANA_URL}
      SERVER_NAME: kibana

      SERVER_PUBLICBASEURL: ${KIBANA_URL}
      # Elasticsearch settings
      ELASTICSEARCH_HOSTS: ${ELASTIC_URL}
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: config/ca.crt
      ELASTICSEARCH_SSL_VERIFICATIONMODE: certificate

      # Elasticsearch monitoring settings
      MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED: true

      # X-Pack Security
      XPACK_SECURITY_ENABLED: true
      XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: ${ENCRYPTION_KEY}
      XPACK_SECURITY_ENCRYPTIONKEY: ${ENCRYPTION_KEY}
      XPACK_REPORTING_ENCRYPTIONKEY: ${ENCRYPTION_KEY}
      XPACK_REPORTING_ROLES_ENABLED: false
      XPACK_REPORTING_KIBANASERVER_HOSTNAME: localhost

      # SSL settings
      SERVER_SSL_ENABLED: true
      SERVER_SSL_CERTIFICATE: config/kibana.crt
      SERVER_SSL_KEY: config/kibana.key
      SERVER_SSL_CERTIFICATEAUTHORITIES: config/ca.crt

      # fleet settings for Elastic-agent
      XPACK_FLEET_ENABLED: true
      XPACK_FLEET_AGENTS_ENABLED: true
      XPACK_FLEET_AGENTS_TLSCHECKDISABLED: true
      XPACK_FLEET_AGENTS_KIBANA_HOST: ${KIBANA_URL}
      XPACK_FLEET_AGENTS_ELASTICSEARCH_HOST: ${ELASTIC_URL}

      #xpack.ingestManager.enabled: true
      #xpack.ingestManager.fleet.enabled: true
      VIRTUAL_HOST: kibana.fire.fly
      VIRTUAL_PORT: 5601
      VIRTUAL_PROTO: https
      LETSENCRYPT_HOST: kibana.fire.fly
    volumes:
      - KB_data:${KIBANA_DIR}/data
    secrets:
      - source: ca.crt
        target: ${KIBANA_DIR}/config/ca.crt
      - source: kibana.cert
        target: ${KIBANA_DIR}/config/kibana.crt
      - source: kibana.key
        target: ${KIBANA_DIR}/config/kibana.key
    ports:
      - "5601:5601"
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
    networks:
      - elk
    depends_on:
      - elasticsearch

  fleet-server:
    container_name: elk_fleet-server
    hostname: fleet-server
    build:
      context: fleet-server/
      args:
        ELK_VERSION: $ELK_VERSION
    restart: always
    environment:
      FLEET_SERVER_ENABLE: true
      FLEET_SERVER_ELASTICSEARCH_HOST: https://elasticsearch:9200
      FLEET_SERVER_SERVICE_TOKEN: ${FLEET_SERVICE_TOKEN}
      FLEET_SERVER_POLICY_NAME: fleet-server-policy
      FLEET_SERVER_ELASTICSEARCH_CA: /ca.pem
      FLEET_SERVER_CERT: ${FILEBEAT_DIR}/config/fleet-server.crt
      FLEET_SERVER_CERT_KEY: ${FILEBEAT_DIR}/config/fleet-server.key
      FLEET_URL: 'https://fleet-server:8220'
      FLEET_CA: /ca.pem
      KIBANA_FLEET_CA: /ca.pem
      ELASTICSEARCH_CA: /ca.pem

      VIRTUAL_HOST: fleet-server.fire.fly
      VIRTUAL_PORT: 8220
      VIRTUAL_PROTO: https
      LETSENCRYPT_HOST: fleet-server.fire.fly
    secrets:
      - source: ca.crt
        target: /ca.pem
      - source: fleet-server.cert
        target: ${FILEBEAT_DIR}/config/fleet-server.crt
      - source: fleet-server.key
        target: ${FILEBEAT_DIR}/config/fleet-server.key
    ports:
      - "8220:8220"
    networks:
      - elk
    depends_on:
      - elasticsearch
      - kibana

networks:
  elk:
    driver: bridge
  dmz:
    external: true
    name: dmz_vlan

When starting the fleet-server container I get the following error message:

{"log.level":"error","@timestamp":"2023-12-17T11:35:18.486Z","message":"Failed to connect to backoff(elasticsearch(https://elasticsearch:9200)): Get \"https://elasticsearch:9200\": x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"Elastic Certificate Tool Autogenerated CA\")","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Basically it says that there is an issue with the CA certificate. I assume that there is some configuration missing to tell the agent which CA certificate he should trust.

Therefore I tried to add all CA configurations to my compose file:

      FLEET_SERVER_ELASTICSEARCH_CA: /ca.pem
      FLEET_CA: /ca.pem
      KIBANA_FLEET_CA: /ca.pem
      ELASTICSEARCH_CA: /ca.pem

I also verified that the /ca.pem is the correct CAfile:

elastic-agent@fleet-server:~$ openssl s_client -connect elasticsearch:9200 -quiet
Can't use SSL_get_servername
depth=0 CN = elasticsearch
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = elasticsearch
verify error:num=21:unable to verify the first certificate
verify return:1



elastic-agent@fleet-server:~$ openssl s_client -connect elasticsearch:9200 -CAfile /ca.pem -quiet
Can't use SSL_get_servername
depth=1 CN = Elastic Certificate Tool Autogenerated CA
verify return:1
depth=0 CN = elasticsearch
verify return:1

I read quite a lot of documentation:

  1. Elastic Agent environment variables | Fleet and Elastic Agent Guide [8.11] | Elastic
  2. Configure SSL/TLS for standalone Elastic Agents | Fleet and Elastic Agent Guide [8.11] | Elastic
  3. Run Elastic Agent in a container | Fleet and Elastic Agent Guide [8.11] | Elastic
  4. Whatever I could find on Google

What am I missing here? Is there another setting to instruct the agent to use my CA file?

Ups - interesting - just looked into kibana and interestingly the fleet-server is reported as healthy.

But I still wondering where these error messages are coming from.

I suspect those errors are coming from the monitoring output for elastic agent. As you probably know, at this point, fleet is really just an elastic agent.

I suspect the monitoring output is not configured properly with the CA.

You can set the monitoring output to use the default I believe. Then I would think it would pick up the rest of those SSL settings.

Set the monitoring output to use the default.

You could probably test by just disabling the monitoring.

Your are right, it is related to the monitoring outputs.

When I change the configuration in the fleet-server policy they are reflected in the error message.

Do you know the environment variable I need to configure in my docker compose that matricbeat (I suppose this ist the part we are talking about) is using my CA cert?

Or is there a way to configure it centrally in kibana?

Just found a solution. I installed the CA certificate in the container itself.
This solved that issue.

However, wouldn't it be much easier to have all elk services rely on the "OS" certificate store for trusted root CAs?

Anyway - thank you very much for your support @stephenb

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.