Kibana cannot reach epr.elastic.co even though "chainpem" is set

hi,

as described here, we have set the environment variable to our “chainpem” likely correctly.

File: /usr/share/kibana/bin/kibana

(see last line)

#!/bin/sh
SCRIPT=$0

# SCRIPT may be an arbitrarily deep series of symlinks. Loop until we have the concrete path.
while [ -h "$SCRIPT" ] ; do
  ls=$(ls -ld "$SCRIPT")
  # Drop everything prior to ->
  link=$(expr "$ls" : '.*-> \(.*\)$')
  if expr "$link" : '/.*' > /dev/null; then
    SCRIPT="$link"
  else
    SCRIPT=$(dirname "$SCRIPT")/"$link"
  fi
done

DIR="$(dirname "${SCRIPT}")/.."
CONFIG_DIR=${KBN_PATH_CONF:-"$DIR/config"}

test_node() {
  test -x "$NODE"
  if [ ! -x "$NODE" ]; then
    echo "unable to find usable node.js executable."
    exit 1
  fi
}

if [ -f "${CONFIG_DIR}/node.options" ]; then
  KBN_NODE_OPTS="$(grep -v ^# < ${CONFIG_DIR}/node.options | xargs)"
fi

NODE="${DIR}/node/glibc-217/bin/node"
test_node

# Enable capturing heap snapshots. See https://nodejs.org/api/cli.html#--heapsnapshot-signalsignal
# For now only on POSIX platforms, term signals work differently on Windows.
NODE_OPTIONS_HEAPSNAPSHOT_DEFAULT="${NODE_OPTIONS_HEAPSNAPSHOT_DEFAULT:---heapsnapshot-signal=SIGUSR2 --diagnostic-dir=$DIR/data}"

NODE_OPTIONS="--no-warnings --max-http-header-size=65536 $NODE_OPTIONS_HEAPSNAPSHOT_DEFAULT $KBN_NODE_OPTS $NODE_OPTIONS" NODE_ENV=production exec "${NODE}" "${DIR}/src/cli/dist" "${@}"
NODE_EXTRA_CA_CERTS="/etc/kibana/CA_Bundle_2024.pem"

I estimate that KIBANA has another cause not to be able to reach epr.elastic.co.

From the operating system, it works:

[300 bin]# curl -vvI "https://epr.elastic.co/search?package=system&internal=true&experimental=true&kibana.version=7.16.1"
*   Trying 34.120.127.130:443...
* Connected to epr.elastic.co (34.120.127.130) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=elastic.co
*  start date: Jul 28 08:55:05 2025 GMT
*  expire date: Oct 26 08:55:04 2025 GMT
*  subjectAltName: host "epr.elastic.co" matched cert's "epr.elastic.co"
*  issuer: C=DE; ST=BW; L=xxxxxxxx; O=xxxxxxxx; OU=IT-Services; CN=XXX Firewall SUB-CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* Using Stream ID: 1 (easy handle 0x5647bba9f7c0)
* TLSv1.2 (OUT), TLS header, Unknown (23):
> HEAD /search?package=system&internal=true&experimental=true&kibana.version=7.16.1 HTTP/2
> Host: epr.elastic.co
> user-agent: curl/7.76.1
> accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
< HTTP/2 200
HTTP/2 200
< date: Thu, 07 Aug 2025 08:39:42 GMT
date: Thu, 07 Aug 2025 08:39:42 GMT
< content-type: application/json
content-type: application/json
< vary: Accept-Encoding
vary: Accept-Encoding
< access-control-allow-origin: *
access-control-allow-origin: *
< cache-control: max-age=600,public
cache-control: max-age=600,public
< strict-transport-security: max-age=31536000; includeSubDomains
strict-transport-security: max-age=31536000; includeSubDomains
< via: 1.1 google
via: 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000


Do I remember correctly that xpack-security has to be enabled to be allowed to activate Fleet or access the epr.elastic.co?

# Enable security features
#xpack.security.enabled: true
#xpack.security.enrollment.enabled: true
#xpack.security.authc.api_key.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["alma92vorlage"]
discovery.type: single-node

Please confirm, if xpack-security is also a requirement for Fleet Function and Integrations!

Regards

EI

I have activated them, no change …

The most helpful thing you can provide are the kibana logs where the epr is trying to connect. I think that would be most helpful.

The certs on the epr.elastic.co are formal, publicly signed certs as shown by your curl... so they should work without you presenting any certs... it is unclear to me why you are trying to do that.

Also Yes I believe you need

elasticsearch.yml

xpack.security.enabled: true

In Kibana I think you need to have

Hi Stephen,

thank you for your reply.

some logs of today morning:

{"cluster_uuid":"WS1AYY9STBqJA9QR4hM2ig","cluster_name":"atlassian-monitoring","error":{"message":"self-signed certificate in certificate chain","name":"Error","stack":"Error: self-signed certificate in certificate chain\n    at Function.AxiosError.from (/usr/share/kibana/node_modules/axios/dist/node/axios.cjs:857:14)\n    at RedirectableRequest.handleRequestError (/usr/share/kibana/node_modules/axios/dist/node/axios.cjs:3169:25)\n    at RedirectableRequest.emit (node:events:530:35)\n    at ClientRequest.eventHandlers.<computed> (/usr/share/kibana/node_modules/follow-redirects/index.js:38:24)\n    at ClientRequest.emit (node:events:518:28)\n    at emitErrorEvent (node:_http_client:104:11)\n    at TLSSocket.socketErrorListener (node:_http_client:518:5)\n    at TLSSocket.emit (node:events:518:28)\n    at emitErrorNT (node:internal/streams/destroy:170:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:129:3)\n    at processTicksAndRejections (node:internal/process/task_queues:90:21)\n    at Axios.request (/usr/share/kibana/node_modules/axios/dist/node/axios.cjs:4258:41)\n    at processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at TelemetryEventsSender.isTelemetryServicesReachable (/usr/share/kibana/node_modules/@kbn/security-solution-plugin/server/lib/telemetry/sender.js:175:20)\n    at SecurityTelemetryTask.runTask (/usr/share/kibana/node_modules/@kbn/security-solution-plugin/server/lib/telemetry/task.js:107:44)\n    at Object.run (/usr/share/kibana/node_modules/@kbn/security-solution-plugin/server/lib/telemetry/task.js:50:30)\n    at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:336:22)","config":{"transitional":{"silentJSONParsing":true,"forcedJSONParsing":true,"clarifyTimeoutError":false},"adapter":["xhr","http","fetch"],"transformRequest":[null],"transformResponse":[null],"timeout":3000,"xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN","maxContentLength":-1,"maxBodyLength":-1,"env":{},"headers":{"Accept":"application/json, text/plain, */*","User-Agent":"axios/1.8.3","Accept-Encoding":"gzip, compress, deflate, br"},"method":"get","url":"https://telemetry.elastic.co/ping","allowAbsoluteUrls":true},"code":"SELF_SIGNED_CERT_IN_CHAIN"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-08T09:13:28.666+02:00","message":"Error pinging telemetry services","log":{"level":"WARN","logger":"plugins.securitySolution.telemetry_events.sender"},"process":{"pid":24219,"uptime":57549.021563392},"trace":{"id":"e7443dfde0916ea422bdbc2e3646c1d9"},"transaction":{"id":"96e0c60956f63992"}}
{"cluster_uuid":"WS1AYY9STBqJA9QR4hM2ig","cluster_name":"atlassian-monitoring","taskId":"security:endpoint-diagnostics:1.1.0","service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-08T09:13:28.666+02:00","message":"Cannot reach telemetry services","log":{"level":"INFO","logger":"plugins.securitySolution.telemetry_events.sender.task"},"process":{"pid":24219,"uptime":57549.02191799},"trace":{"id":"e7443dfde0916ea422bdbc2e3646c1d9"},"transaction":{"id":"96e0c60956f63992"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-08T09:13:42.861+02:00","message":"Error connecting to package registry: request to https://epr.elastic.co/categories?kibana.version=9.1.1&spec.min=2.3&spec.max=3.4 failed, reason: self-signed certificate in certificate chain","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":24219,"uptime":57563.217663859},"trace":{"id":"e1b8602686d110d85ade9e2c72bdf55b"},"transaction":{"id":"03a74948181f8450"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-08T09:13:42.867+02:00","message":"Error connecting to package registry: request to https://epr.elastic.co/search?kibana.version=9.1.1&spec.min=2.3&spec.max=3.4 failed, reason: self-signed certificate in certificate chain","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":24219,"uptime":57563.222475144},"trace":{"id":"0ca11ffd4feb918402c21757e2ba33c7"},"transaction":{"id":"6309ecd89e2a2c3a"}}

The cert is requested because of our forward proxy. The Linux host itself can access the destination, however JDK of ELK has issues with it.

The only solution I found was this option:

NODE_EXTRA_CA_CERTS="/etc/kibana/CA_Bundle_2024.pem"

It did work neither …

Are those logs from Kibana?

Did you register the proxy?

BTW minor, Kibana is nodejs not Java

Also Others did get this to work with NODE_EXTRA_CA_CERTS

Did you try running like this. And made sure that file is readable by Kibana user / permissions are correct

NODE_EXTRA_CA_CERTS="/etc/kibana/CA_Bundle_2024.pem" /usr/share/kibana/bin/kibana

Thank you Stephen,

the file “kibana” is owned by root

The certificate himself can be read by the group KIBANA, it should be OK.

Concerning this

docker run --name kibana \ --net elastic \ -v $(pwd)/npmcerts.pem:/tmp/npmcerts.pem \ -e NODE_EXTRA_CA_CERTS=/tmp/npmcerts.pem \ -p 5601:5601 \ docker.elastic.co/kibana/kibana:8.0.0

I do not use DOCKER, I have simply copied the line in the file “kibana”.

For any reason KIBANA does not recognise our root CERT and throws error because of self-signed certificates.

Perhaps anything goes wrong at this point?

I understand you're not using docker. I was just showing that example.

Did you try starting exactly like this with a path to your pem

NODE_EXTRA_CA_CERTS="/etc/kibana/CA_Bundle_2024.pem" /usr/share/kibana/bin/kibana

I would also make sure it's readable by root and kibana just to check you can tighten it down later.

Also, did you register the proxy?

Hi Stephen,

thank you! This part has been forgotten.

  • After adding it to the config and restarting alle services, it didn´t worked yet.
  • Concerning the proxy, it is not a real forward proxy but a so called “transparent proxy” of our Firewall and has the job to exchange the certificate.
  • Regarding the permissions the cert is owned by root and belongs to group KIBANA. Should be OK.

Regards

EI

@Elasticisti

I am not sure what to tell you at this point...

Not really familiar with that... something is obviously not working.

Did you register the proxy?

You also did not share the Kibana Logs... there may be something helpful there

Hi stephen,

some logs.

Error: self-signed certificate in certificate chain\n
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T18:00:51.371+02:00","message":"[runTask()] started","log":{"level":"INFO","logger":"plugins.fleet.fleet:sync-integrations-task:1.0.5"},"process":{"pid":2544347,"uptime":3584.649530499},"trace":{"id":"e7443dfde0916ea422bdbc2e3646c1d9"},"transaction":{"id":"3ffc1fd2142d6597"}}
{"cluster_uuid":"WS1AYY9STBqJA9QR4hM2ig","cluster_name":"atlassian-monitoring","error":{"message":"self-signed certificate in certificate chain","name":"Error","stack":"Error: self-signed certificate in certificate chain\n    at Function.AxiosError.from (/usr/share/kibana/node_modules/axios/dist/node/axios.cjs:857:14)\n    at RedirectableRequest.handleRequestError (/usr/share/kibana/node_modules/axios/dist/node/axios.cjs:3169:25)\n    at RedirectableRequest.emit (node:events:530:35)\n    at ClientRequest.eventHandlers.<computed> (/usr/share/kibana/node_modules/follow-redirects/index.js:38:24)\n    at ClientRequest.emit (node:events:518:28)\n    at emitErrorEvent (node:_http_client:104:11)\n    at TLSSocket.socketErrorListener (node:_http_client:518:5)\n    at TLSSocket.emit (node:events:518:28)\n    at emitErrorNT (node:internal/streams/destroy:170:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:129:3)\n    at processTicksAndRejections (node:internal/process/task_queues:90:21)\n    at Axios.request (/usr/share/kibana/node_modules/axios/dist/node/axios.cjs:4258:41)\n    at processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at TelemetryEventsSender.isTelemetryServicesReachable (/usr/share/kibana/node_modules/@kbn/security-solution-plugin/server/lib/telemetry/sender.js:175:20)\n    at SecurityTelemetryTask.runTask (/usr/share/kibana/node_modules/@kbn/security-solution-plugin/server/lib/telemetry/task.js:107:44)\n    at Object.run (/usr/share/kibana/node_modules/@kbn/security-solution-plugin/server/lib/telemetry/task.js:50:30)\n    at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:336:22)","config":{"transitional":{"silentJSONParsing":true,"forcedJSONParsing":true,"clarifyTimeoutError":false},"adapter":["xhr","http","fetch"],"transformRequest":[null],"transformResponse":[null],"timeout":3000,"xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN","maxContentLength":-1,"maxBodyLength":-1,"env":{},"headers":{"Accept":"application/json, text/plain, */*","User-Agent":"axios/1.8.3","Accept-Encoding":"gzip, compress, deflate, br"},"method":"get","url":"https://telemetry.elastic.co/ping","allowAbsoluteUrls":true},"code":"SELF_SIGNED_CERT_IN_CHAIN"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T18:00:51.400+02:00","message":"Error pinging telemetry services","log":{"level":"WARN","logger":"plugins.securitySolution.telemetry_events.sender"},"process":{"pid":2544347,"uptime":3584.678940846},"trace":{"id":"e7443dfde0916ea422bdbc2e3646c1d9"},"transaction":{"id":"0cd69122ffa7580f"}}
{"cluster_uuid":"WS1AYY9STBqJA9QR4hM2ig","cluster_name":"atlassian-monitoring","taskId":"security:endpoint-diagnostics:1.1.0","service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T18:00:51.401+02:00","message":"Cannot reach telemetry services","log":{"level":"INFO","logger":"plugins.securitySolution.telemetry_events.sender.task"},"process":{"pid":2544347,"uptime":3584.679290826},"trace":{"id":"e7443dfde0916ea422bdbc2e3646c1d9"},"transaction":{"id":"0cd69122ffa7580f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T18:03:30.359+02:00","message":"[runTask()] started","log":{"level":"INFO","logger":"plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.1"},"process":{"pid":2544347,"uptime":3743.637511323},"trace":{"id":"e7443dfde0916ea422bdbc2e3646c1d9"},"transaction":{"id":"6cf8b8870e959f18"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T18:03:30.372+02:00","message":"[UnenrollInactiveAgentsTask] runTask ended: success","log":{"level":"INFO","logger":"plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.1"},"process":{"pid":2544347,"uptime":3743.650455683},"trace":{"id":"e7443dfde0916ea422bdbc2e3646c1d9"},"transaction":{"id":"6cf8b8870e959f18"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T18:03:45.986+02:00","message":"Secrets storage is disabled as minimum fleet server version has not been met","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":2544347,"uptime":3759.26507134},"trace":{"id":"ba8a3bfdd8c01d4edfcdf7e5f17af059"},"transaction":{"id":"374ec14a2e25171e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T18:04:17.309+02:00","message":"Error connecting to package registry: request to https://epr.elastic.co/search?kibana.version=9.1.1&spec.min=2.3&spec.max=3.4 failed, reason: self-signed certificate in certificate chain","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":2544347,"uptime":3790.588204287},"trace":{"id":"8d72a2b187db7d28a51bda5f172ce646"},"transaction":{"id":"8308552fef6bf44f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T18:04:17.335+02:00","message":"Error connecting to package registry: request to https://epr.elastic.co/categories?kibana.version=9.1.1&spec.min=2.3&spec.max=3.4 failed, reason: self-signed certificate in certificate chain","log":{"level":"ERROR","logger":"plugins.fleet"},"process":{"pid":2544347,"uptime":3790.614060666},"trace":{"id":"55fbd0057fc1e6a7257bcb413c080af7"},"transaction":{"id":"91281e0dd461b6ab"}}

Concerning Proxy I will ask the colleagues being responsible for the firewall ..#

Thank you anyway for your support. We appreciate it.

Regards

EI

1 Like

Did you just try putting this in your kibana.yml

xpack.fleet.registryProxyUrl: your-nat-gateway.corp.net

Hmm, If I add this Line, I simply cannot start KIBANA:

KIBANA.YML

server.host: 0.0.0.0

server.name: "atlassian-monitoring"

#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid

# This section was automatically generated during setup.
elasticsearch.hosts: [https://172.30.68.65:9200]
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE3NTQ1NzU4MDEzOTc6YlFiSjIweUdUUmFoaUxxxxxxxxxxxx
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1754575802535.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: [https://172.30.68.65:9200], ca_trusted_fingerprint: eef54df23ee933914ed2289595dcd2a1122c96eb2ad52ad4d15037xxxxxxxxxxx}]
xpack.encryptedSavedObjects.encryptionKey: e64a29e1f677f97d60678150xxxxxx
xpack.fleet.registryProxyUrl: your-nat-gateway.corp.net

I think

your-nat-gateway.corp.net

should be replaced with the real URL, which I have to clear with the staff.

LOG:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:10.358+02:00","message":"SIGTERM received - initiating shutdown","log":{"level":"INFO","logger":"root"},"process":{"pid":2544347,"uptime":8703.636928299}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:10.359+02:00","message":"Kibana is shutting down","log":{"level":"INFO","logger":"root"},"process":{"pid":2544347,"uptime":8703.637528487}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:10.366+02:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":2544347,"uptime":8703.644713593}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:10.370+02:00","message":"Un-registering task definition [endpoint:complete-external-response-actions] (if it exists)","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:complete-external-response-actions"},"process":{"pid":2544347,"uptime":8703.649082125}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:10.372+02:00","message":"Monitoring stats collection is stopped","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":2544347,"uptime":8703.650975291}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:10.377+02:00","message":"Stopping the task poller","log":{"level":"INFO","logger":"plugins.taskManager"},"process":{"pid":2544347,"uptime":8703.655247555}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:10.532+02:00","message":"Removed this node from the Kibana Discovery Service","log":{"level":"INFO","logger":"plugins.taskManager"},"process":{"pid":2544347,"uptime":8703.810448205}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:10.532+02:00","message":"All plugins stopped.","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":2544347,"uptime":8703.811177211}}
{"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:31.849+02:00","message":"Kibana is starting","log":{"level":"INFO","logger":"root"},"process":{"pid":2607500,"uptime":21.044248395}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:26:31.926+02:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":2607500,"uptime":21.072488112},"trace":{"id":"d50d736180883551a1fff7a4b36e4a75"},"transaction":{"id":"d3179076f486aff2"}}
{"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:27:12.018+02:00","message":"Kibana is starting","log":{"level":"INFO","logger":"root"},"process":{"pid":2607972,"uptime":4.218062466}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:27:12.063+02:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":2607972,"uptime":4.248225266},"trace":{"id":"c2fcc79632f220f76c58142ee8a149c4"},"transaction":{"id":"0ff95ed42498d974"}}
{"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:27:24.908+02:00","message":"Kibana is starting","log":{"level":"INFO","logger":"root"},"process":{"pid":2608112,"uptime":4.111550258}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:27:24.945+02:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":2608112,"uptime":4.1370836},"trace":{"id":"c5f56280d9cc6ffb928f96e16af18f38"},"transaction":{"id":"f374ce74ec87cfb6"}}
{"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:27:38.279+02:00","message":"Kibana is starting","log":{"level":"INFO","logger":"root"},"process":{"pid":2608203,"uptime":4.233646393}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:27:38.314+02:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":2608203,"uptime":4.25393359},"trace":{"id":"2c6cbf0b3f62f3b4aeb4b5d074ad7773"},"transaction":{"id":"82434777b1c7039e"}}
{"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:27:51.594+02:00","message":"Kibana is starting","log":{"level":"INFO","logger":"root"},"process":{"pid":2608285,"uptime":4.297194252}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:27:51.626+02:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":2608285,"uptime":4.317538456},"trace":{"id":"d96bfa902ea125bac4fb5010c0aae7a0"},"transaction":{"id":"a07a8e3d15ab8f46"}}
{"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:39:48.083+02:00","message":"Kibana is starting","log":{"level":"INFO","logger":"root"},"process":{"pid":2613891,"uptime":4.252897072}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:39:48.118+02:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":2613891,"uptime":4.273605664},"trace":{"id":"900b871a313723eb181603ced93d693b"},"transaction":{"id":"1d32df56207ee4ea"}}
{"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:40:01.020+02:00","message":"Kibana is starting","log":{"level":"INFO","logger":"root"},"process":{"pid":2613980,"uptime":4.224860082}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:40:01.057+02:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":2613980,"uptime":4.249959755},"trace":{"id":"a92c1fc7bad82cdf1121200a8f1e4e25"},"transaction":{"id":"5662bb329f45966a"}}
{"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:40:14.233+02:00","message":"Kibana is starting","log":{"level":"INFO","logger":"root"},"process":{"pid":2614120,"uptime":4.179296012}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2025-08-11T19:40:14.269+02:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":2614120,"uptime":4.204403106},"trace":{"id":"17ffd7f60a45d0f2165bb1c44010c6a6"},"transaction":{"id":"af46e10adacdee37"}}

Yes.... That's just an example

That may or may not fix... I'm not sure really what a transparent proxy is

So, your firewall is opening the ssl connections to inspect the packages, basically acting as MITM.

In this case you need to make Kibana (which is a node application) to recognize your CA, but just adding NODE_EXTRA_CA_CERTS="/etc/kibana/CA_Bundle_2024.pem" in the startup script will not do that, it just create a variable with this name and value, but it does not export it as an environment variable for the Kibana process.

Are you running Kibana as a service?

Thank you.

yeah, this is exactly the case …

Regards

EI

I think you need to load the environment variable through the systemd service.

The kibana service file will load environment variables from a file that can be found in 2 locations, which one will depend on your distribution (readhat or debian based).

These are the locations:

EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana

Add the line NODE_EXTRA_CA_CERTS="/etc/kibana/CA_Bundle_2024.pem" to one of those files, depending on your operating system, and restart the kibana service to see if it works.

Also, remove the proxy configurations, I don't think you can use them as you do not seem to have a proxy, but a firewall working as a MITM.

If your certificate bundle is correct and has the CA of the certificate that the firewall is replacing, it should work.

1 Like

@leandrojmp now it works :star_struck:

@stephenb thank you also for your time and efforts.

Guys, I would like to slide a beer across the bar, but there would be logistical problems. So simply, thank you!

3 Likes