Fleet Server unhealthy in GUI and from CLI.
Hello All,
I have been wrestling with adding the fleet server for sometime now. I am running self managed ELK with 3 nodes as follows.
master/data - Also runs kibana
data01
date02
I used the command below to install fleet server. My kibana and elasticsearch install uses the self signed TLS certificates generated by elasticsearch-certutil. fleet certificates where generated by the same tool and use the same CA. This piece actually works. The fleet install also works but does not complete fully.
See the status messages below.
status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
This sort of message usually means that 'elasticsearch.ssl.certificate' is set up, somewhere, with multiple values and with square brackets [ and ]. This confuses the elastic-agent. The question is where is the certificate set? Which file / location? How can it be changed without breaking other processes that are already using the certificates?
[root@rhoslog01 elastic-agent-9db552]# elastic-agent status
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a failed state
├─ beat/metrics-monitoring
│ ├─ status: (HEALTHY) Healthy: communicating with pid '16899'
│ ├─ beat/metrics-monitoring
│ │ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
│ └─ beat/metrics-monitoring-metrics-monitoring-beats
│ └─ status: (STARTING) Starting
├─ filestream-monitoring
│ ├─ status: (HEALTHY) Healthy: communicating with pid '16888'
│ ├─ filestream-monitoring
│ │ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
│ └─ filestream-monitoring-filestream-monitoring-agent
│ └─ status: (STARTING) Starting
└─ http/metrics-monitoring
├─ status: (HEALTHY) Healthy: communicating with pid '16876'
├─ http/metrics-monitoring
│ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
└─ http/metrics-monitoring-metrics-monitoring-agent
└─ status: (STARTING) Starting
[root@rhoslog01 elastic-agent-9db552]#
Below is my install command for fleet server
./elastic-agent install --url=https://XXXXlog01.DOMAIN1.HHH.CC:8220 --fleet-server-es=https://XXXXlog01.DOMAIN1.HHH.CC:9200 --fleet-server-service-token=CHANGEDCHANGEDaWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MTEwMzkzMjI2MDg6SnlfSVJoUm1TZ1dILWVoc1dhM0lEZw --fleet-server-policy=fleet-server-policy --fleet-server-es-ca=/etc/elasticsearch/certs/elastic-stack-ca.crt --fleet-server-cert=/etc/elasticsearch/certs/fleet-server/fleet-server.crt --fleet-server-cert-key=/etc/elasticsearch/certs/fleet-server/fleet-server.key --certificate-authorities=/etc/elasticsearch/certs/elastic-stack-ca.crt --fleet-server-port=8220 --fleet-server-host=XXXXlog01.DOMAIN1.HHH.CC
I have changed the hostnames and the service-token but they are normal / correct when I run the command. To be clear I am just doing the install on my master node.
I can curl to the fleet server url, kibana url and the elasticsearch url from that server.
curl -kv https://rhoslog01.agriculture.gov.ie:9200
* Rebuilt URL to: https://rhoslog01.agriculture.gov.ie:9200/
* Uses proxy env variable no_proxy == '.agriculture.gov.ie,10.0.0.1,10.0.0.2,10.0.0.3,.cluster.local,.svc,localhost,127.0.0.1,172.30.0.1'
* Trying 10.2.134.121...
* TCP_NODELAY set
* Connected to rhoslog01.agriculture.gov.ie (10.2.134.121) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=master-01
* start date: Oct 25 12:56:22 2022 GMT
* expire date: Oct 24 12:56:22 2025 GMT
* issuer: CN=Elastic Certificate Tool Autogenerated CA
* SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET / HTTP/1.1
> Host: rhoslog01.agriculture.gov.ie:9200
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< WWW-Authenticate: Bearer realm="security"
< WWW-Authenticate: ApiKey
< content-type: application/json
< content-length: 459
<
* Connection #0 to host rhoslog01.agriculture.gov.ie left intact
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}[root@rhoslog01 elastic-agent-9db552]
curl -kv https://rhoslog01.agriculture.gov.ie:8220
* Rebuilt URL to: https://rhoslog01.agriculture.gov.ie:8220/
* Uses proxy env variable no_proxy == '.agriculture.gov.ie,10.0.0.1,10.0.0.2,10.0.0.3,.cluster.local,.svc,localhost,127.0.0.1,172.30.0.1'
* Trying 10.2.134.121...
* TCP_NODELAY set
* Connected to rhoslog01.agriculture.gov.ie (10.2.134.121) port 8220 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=fleet-server
* start date: Mar 12 14:54:58 2024 GMT
* expire date: Mar 12 14:54:58 2027 GMT
* issuer: CN=Elastic Certificate Tool Autogenerated CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* Using Stream ID: 1 (easy handle 0x55cf124726d0)
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET / HTTP/2
> Host: rhoslog01.agriculture.gov.ie:8220
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/2 404
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< x-request-id: 7c83296b-3f8c-419f-9011-ea36e8046f58
< content-length: 19
< date: Fri, 22 Mar 2024 14:41:47 GMT
<
* TLSv1.3 (IN), TLS app data, [no content] (0):
404 page not found
* Connection #0 to host rhoslog01.agriculture.gov.ie left intact
[root@rhoslog01 elastic-agent-9db552]#
curl -kv http://rhoslog01.agriculture.gov.ie:5601
* Rebuilt URL to: http://rhoslog01.agriculture.gov.ie:5601/
* Uses proxy env variable no_proxy == '.agriculture.gov.ie,10.0.0.1,10.0.0.2,10.0.0.3,.cluster.local,.svc,localhost,127.0.0.1,172.30.0.1'
* Trying 10.2.134.121...
* TCP_NODELAY set
* Connected to rhoslog01.agriculture.gov.ie (10.2.134.121) port 5601 (#0)
> GET / HTTP/1.1
> Host: rhoslog01.agriculture.gov.ie:5601
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 302 Found
< location: /login?next=%2F
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< permissions-policy: camera=(), display-capture=(), fullscreen=(self), geolocation=(), microphone=(), web-share=()
< cross-origin-opener-policy: same-origin
< content-security-policy: script-src 'report-sample' 'self'; worker-src 'report-sample' 'self' blob:; style-src 'report-sample' 'self' 'unsafe-inline'
< kbn-name: kibana
< kbn-license-sig: f0013dee1fb80e542679f4d77da090bb67f855801d3f263ec6cf88a4c439ed7d
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 0
< Date: Fri, 22 Mar 2024 14:42:30 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
* Connection #0 to host rhoslog01.agriculture.gov.ie left intact
So my question is what certificates do the fleet components / elastic-agent use that cannot connect due to "unpacking 'ssl' config: can not convert 'object' into 'string'" problem above?
Is it in /etc/kibana/kibana.yml, /etc/elasticsearch/elasticsearch.yml or in the various yml files in /opt/Elastic/Agent/data/elastic-agent-9db552/components ?
Below is results of elastic-agent inspect command.
Any help much appreciated.
If I have not made anything clear please ask me to provided more details.
Best Regards,
Kevin.
elastic-agent inspect
agent:
download:
sourceURI: https://artifacts.elastic.co/downloads/
features: null
id: e9599bf7-a5ac-4b2a-94ad-4c9a3c481963
monitoring:
enabled: true
http:
buffer: null
enabled: false
host: localhost
port: 6791
logs: true
metrics: true
namespace: default
use_output: default
protection:
enabled: false
signing_key: MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEhmDF64wHjP4cbXoCNwkl16IoGlJ5GFS9mDVvQJp/VMyiLqkkDznz7f4srOpPzpBKSBxgPbifuBXM5iPdto7ZwA==
uninstall_token_hash: DZPSZoOK9j7xUqKsdRQ0XYLpFDtO02Y3d+Pa1cwcobE=
fleet:
access_api_key: VTZYa1lZNEJrUVFpQ1Qtazd4RGs6OUR0T3FsN2dSUnFwVFQxS2lVaDFodw==
agent:
id: ""
enabled: true
host: localhost:8221
hosts:
- https://10.2.134.121:8220
protocol: https
proxy_disable: true
server:
host: rhoslog01.agriculture.gov.ie
internal_port: 8221
output:
elasticsearch:
hosts:
- rhoslog01.agriculture.gov.ie:9200
protocol: https
proxy_disable: false
proxy_headers: null
service_token: AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MTEwMzkzMjI2MDg6SnlfSVJoUm1TZ1dILWVoc1dhM0lEZw
ssl:
certificate_authorities:
- /etc/elasticsearch/certs/elastic-stack-ca.crt
renegotiation: never
verification_mode: full
policy:
id: fleet-server-policy
port: 8220
ssl:
certificate: /etc/elasticsearch/certs/fleet-server/fleet-server.crt
key: /etc/elasticsearch/certs/fleet-server/fleet-server.key
renegotiation: never
verification_mode: full
ssl:
certificate_authorities:
- /etc/elasticsearch/certs/elastic-stack-ca.crt
renegotiation: never
verification_mode: certificate
timeout: 10m0s
host:
id: c6388f19030e4777b74d2994b30f03da
id: fleet-server-policy
inputs:
- data_stream:
namespace: default
id: fleet-server-fleet_server-355fc92f-6a7b-444e-a10a-7d4027da7362
meta:
package:
name: fleet_server
version: 1.5.0
name: fleet_server-1
package_policy_id: 355fc92f-6a7b-444e-a10a-7d4027da7362
revision: 1
type: fleet-server
unused_key: not_used
use_output: default
output_permissions:
default:
_elastic_agent_checks:
cluster:
- monitor
_elastic_agent_monitoring:
indices:
- names:
- logs-elastic_agent.apm_server-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.apm_server-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.auditbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.auditbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.cloud_defend-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.cloudbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.cloudbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.elastic_agent-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.endpoint_security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.endpoint_security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.filebeat_input-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.filebeat_input-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.filebeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.filebeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.fleet_server-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.fleet_server-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.heartbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.heartbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.metricbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.metricbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.osquerybeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.osquerybeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.packetbeat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-elastic_agent.packetbeat-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.pf_elastic_collector-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.pf_elastic_symbolizer-default
privileges:
- auto_configure
- create_doc
- names:
- logs-elastic_agent.pf_host_agent-default
privileges:
- auto_configure
- create_doc
355fc92f-6a7b-444e-a10a-7d4027da7362:
indices: []
outputs:
default:
api_key: fqXkYY4BkQQiCT-k-xTq:6GNCvGAbTRqapPadVZpOvg
hosts:
- https://10.2.134.121:9200
- https://10.2.134.122:9200
- https://10.2.134.123:9200
preset: balanced
ssl:
ca_trusted_fingerprint: CBEA99D8A59F39F21FBEAA1BD3B2334D620EB408
certificate:
authorities:
- |
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIVANEFkHTzX9WraDrfYzs/1/Thg9PYMA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTIyMTAyNTEyNTUyMFoXDTI1MTAyNDEyNTUyMFowNDEyMDAG
A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDT1ypwRYxBldW++hahcCwG
4oYf/4POg752WmCysQ8RhjJhacp+0bQNTk7RT3Nr2gIu+kGNVU76oDthLtAFg3d/
R4v1GOjfLlUfykJJ8exhGX4hRKAk5dpu/CCuqzGkvCCaezqJCxNnZ8rkWGVcZKC/
a8gzGXZE9uEtDTAWiognXKigeRLoMFu75pHUOUyb2tFOA+GPbmv05EwKbfKMcPF0
qKMSmdRwCCy03lBFbLt52IzXK78xNKyUFT2tWyci/jZbBoeJtd+ypTPh8o8Nz9mg
1vnadhm5uQAl4Rp4iidSUKBdE0FKSd4VlCr2BcvyzvVH/xUF6f8pzN1j0NEiSBF9
AgMBAAGjUzBRMB0GA1UdDgQWBBSNvM4HrTqqZn6cQsBmPbx28shbOzAfBgNVHSME
GDAWgBSNvM4HrTqqZn6cQsBmPbx28shbOzAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQAMD1aL638RkwHMErmfVwxYuakI7VawDZCInS7FEIT+RbhA
XYmODAt1tvRqkVr5+LkBsHsufl/9ZOn8To3/tlW6QQlNixc/B6e5tirrFN5dbfT9
za/tc/34nkzWcdPJ/tJKmelEATZlezzOyR87pXVMLUTZeNOULtLLOTOxhqJGeAof
mQp0eey8BjcvV3j2N9NWA7tE0B4iVm/pR4EtnOpsUrW6VjkHJRVbKaqO+ul9lI11
XYkRQJ4G+h0fqV7qdcqLRzuqmnwUun7K3KdLdY7BuLuJbB947rtpZ9ZrcDMI9wY2
wEwJmstrO3CC9rK3pN1O64xFGxWzMFHC3xqsT1YW
-----END CERTIFICATE-----
type: elasticsearch
path:
config: /opt/Elastic/Agent
data: /opt/Elastic/Agent/data
home: /opt/Elastic/Agent/data/elastic-agent-9db552
logs: /opt/Elastic/Agent
revision: 134
runtime:
arch: amd64
os: linux
osinfo:
family: redhat
major: 8
minor: 8
patch: 0
type: linux
version: 8.8 (Ootpa)
signed:
data: eyJpZCI6ImZsZWV0LXNlcnZlci1wb2xpY3kiLCJhZ2VudCI6eyJmZWF0dXJlcyI6e30sInByb3RlY3Rpb24iOnsiZW5hYmxlZCI6ZmFsc2UsInVuaW5zdGFsbF90b2tlbl9oYXNoIjoiRFpQU1pvT0s5ajd4VXFLc2RSUTBYWUxwRkR0TzAyWTNkK1BhMWN3Y29iRT0iLCJzaWduaW5nX2tleSI6Ik1Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRWhtREY2NHdIalA0Y2JYb0NOd2tsMTZJb0dsSjVHRlM5bURWdlFKcC9WTXlpTHFra0R6bno3ZjRzck9wUHpwQktTQnhnUGJpZnVCWE01aVBkdG83WndBPT0ifX0sImlucHV0cyI6W3siaWQiOiJmbGVldC1zZXJ2ZXItZmxlZXRfc2VydmVyLTM1NWZjOTJmLTZhN2ItNDQ0ZS1hMTBhLTdkNDAyN2RhNzM2MiIsIm5hbWUiOiJmbGVldF9zZXJ2ZXItMSIsInJldmlzaW9uIjoxLCJ0eXBlIjoiZmxlZXQtc2VydmVyIn1dfQ==
signature: MEQCIFTmAm/yOmRPW8HV/2iMfkYihW5zL7kPWWp9IQZzxxICAiAa8Fl5BMa95H7JbiFh3r/XEG+xFBZrvcl23117AY2KzA==
[root@rhoslog01 components]#
