I think the beat components config is derived from the agent policy config. It's good that the certificate_authorities is set in the UI, but it doesn't override the wrong config.
One workaround you could try is to override the incorrect value on the UI Advanced YAML configuration, something like setting null or copy the actual value of the certificate.
ssl.certificate: null
Hi Julia,
I will try that but the thing. One question though is the problem certificate in elasticsearch.ssl.certificate the certificate authority certificate or the actual elasticsearch certificate?
From the files like beat-rendered-config.yml it appears that the authorities certificate is set but in a different way. Is the different way causing the problem? Or the lack of another certificate?
Can the actual deployed (and used) versions of the files like beat-rendered-config.yml be directly manipulated?
Best Regards,
Kevin.
Hi Julia,
I have tried settings as follows.
ssl.certificate_authorities:
- |
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIVANEFkHTzX9WraDrfYzs/1/Thg9PYMA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTIyMTAyNTEyNTUyMFoXDTI1MTAyNDEyNTUyMFowNDEyMDAG
A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDT1ypwRYxBldW++hahcCwG
4oYf/4POg752WmCysQ8RhjJhacp+0bQNTk7RT3Nr2gIu+kGNVU76oDthLtAFg3d/
R4v1GOjfLlUfykJJ8exhGX4hRKAk5dpu/CCuqzGkvCCaezqJCxNnZ8rkWGVcZKC/
a8gzGXZE9uEtDTAWiognXKigeRLoMFu75pHUOUyb2tFOA+GPbmv05EwKbfKMcPF0
qKMSmdRwCCy03lBFbLt52IzXK78xNKyUFT2tWyci/jZbBoeJtd+ypTPh8o8Nz9mg
1vnadhm5uQAl4Rp4iidSUKBdE0FKSd4VlCr2BcvyzvVH/xUF6f8pzN1j0NEiSBF9
AgMBAAGjUzBRMB0GA1UdDgQWBBSNvM4HrTqqZn6cQsBmPbx28shbOzAfBgNVHSME
GDAWgBSNvM4HrTqqZn6cQsBmPbx28shbOzAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQAMD1aL638RkwHMErmfVwxYuakI7VawDZCInS7FEIT+RbhA
XYmODAt1tvRqkVr5+LkBsHsufl/9ZOn8To3/tlW6QQlNixc/B6e5tirrFN5dbfT9
za/tc/34nkzWcdPJ/tJKmelEATZlezzOyR87pXVMLUTZeNOULtLLOTOxhqJGeAof
mQp0eey8BjcvV3j2N9NWA7tE0B4iVm/pR4EtnOpsUrW6VjkHJRVbKaqO+ul9lI11
XYkRQJ4G+h0fqV7qdcqLRzuqmnwUun7K3KdLdY7BuLuJbB947rtpZ9ZrcDMI9wY2
wEwJmstrO3CC9rK3pN1O64xFGxWzMFHC3xqsT1YW
-----END CERTIFICATE-----
ssl.certificate: ["/etc/elasticsearch/certs/elastic-stack-ca.crt"]
This did not fix the problem. See below.
/opt/Elastic/Agent/elastic-agent status
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a failed state
├─ beat/metrics-monitoring
│ ├─ status: (HEALTHY) Healthy: communicating with pid '10059'
│ ├─ beat/metrics-monitoring
│ │ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
│ └─ beat/metrics-monitoring-metrics-monitoring-beats
│ └─ status: (STARTING) Starting
├─ filestream-monitoring
│ ├─ status: (HEALTHY) Healthy: communicating with pid '10049'
│ ├─ filestream-monitoring
│ │ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
│ └─ filestream-monitoring-filestream-monitoring-agent
│ └─ status: (STARTING) Starting
└─ http/metrics-monitoring
├─ status: (HEALTHY) Healthy: communicating with pid '10074'
├─ http/metrics-monitoring
│ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
└─ http/metrics-monitoring-metrics-monitoring-agent
└─ status: (STARTING) Starting
Best Regards,
Kevin.
Hi Julia,
Next I tried this set-up but I still have the same problem unfortunately.
Best Regards,
Kevin.
ssl.certificate_authorities:
- |
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIVANEFkHTzX9WraDrfYzs/1/Thg9PYMA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTIyMTAyNTEyNTUyMFoXDTI1MTAyNDEyNTUyMFowNDEyMDAG
A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDT1ypwRYxBldW++hahcCwG
4oYf/4POg752WmCysQ8RhjJhacp+0bQNTk7RT3Nr2gIu+kGNVU76oDthLtAFg3d/
R4v1GOjfLlUfykJJ8exhGX4hRKAk5dpu/CCuqzGkvCCaezqJCxNnZ8rkWGVcZKC/
a8gzGXZE9uEtDTAWiognXKigeRLoMFu75pHUOUyb2tFOA+GPbmv05EwKbfKMcPF0
qKMSmdRwCCy03lBFbLt52IzXK78xNKyUFT2tWyci/jZbBoeJtd+ypTPh8o8Nz9mg
1vnadhm5uQAl4Rp4iidSUKBdE0FKSd4VlCr2BcvyzvVH/xUF6f8pzN1j0NEiSBF9
AgMBAAGjUzBRMB0GA1UdDgQWBBSNvM4HrTqqZn6cQsBmPbx28shbOzAfBgNVHSME
GDAWgBSNvM4HrTqqZn6cQsBmPbx28shbOzAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQAMD1aL638RkwHMErmfVwxYuakI7VawDZCInS7FEIT+RbhA
XYmODAt1tvRqkVr5+LkBsHsufl/9ZOn8To3/tlW6QQlNixc/B6e5tirrFN5dbfT9
za/tc/34nkzWcdPJ/tJKmelEATZlezzOyR87pXVMLUTZeNOULtLLOTOxhqJGeAof
mQp0eey8BjcvV3j2N9NWA7tE0B4iVm/pR4EtnOpsUrW6VjkHJRVbKaqO+ul9lI11
XYkRQJ4G+h0fqV7qdcqLRzuqmnwUun7K3KdLdY7BuLuJbB947rtpZ9ZrcDMI9wY2
wEwJmstrO3CC9rK3pN1O64xFGxWzMFHC3xqsT1YW
-----END CERTIFICATE-----
ssl.certificate:
- |
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIVANEFkHTzX9WraDrfYzs/1/Thg9PYMA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTIyMTAyNTEyNTUyMFoXDTI1MTAyNDEyNTUyMFowNDEyMDAG
A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDT1ypwRYxBldW++hahcCwG
4oYf/4POg752WmCysQ8RhjJhacp+0bQNTk7RT3Nr2gIu+kGNVU76oDthLtAFg3d/
R4v1GOjfLlUfykJJ8exhGX4hRKAk5dpu/CCuqzGkvCCaezqJCxNnZ8rkWGVcZKC/
a8gzGXZE9uEtDTAWiognXKigeRLoMFu75pHUOUyb2tFOA+GPbmv05EwKbfKMcPF0
qKMSmdRwCCy03lBFbLt52IzXK78xNKyUFT2tWyci/jZbBoeJtd+ypTPh8o8Nz9mg
1vnadhm5uQAl4Rp4iidSUKBdE0FKSd4VlCr2BcvyzvVH/xUF6f8pzN1j0NEiSBF9
AgMBAAGjUzBRMB0GA1UdDgQWBBSNvM4HrTqqZn6cQsBmPbx28shbOzAfBgNVHSME
GDAWgBSNvM4HrTqqZn6cQsBmPbx28shbOzAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQAMD1aL638RkwHMErmfVwxYuakI7VawDZCInS7FEIT+RbhA
XYmODAt1tvRqkVr5+LkBsHsufl/9ZOn8To3/tlW6QQlNixc/B6e5tirrFN5dbfT9
za/tc/34nkzWcdPJ/tJKmelEATZlezzOyR87pXVMLUTZeNOULtLLOTOxhqJGeAof
mQp0eey8BjcvV3j2N9NWA7tE0B4iVm/pR4EtnOpsUrW6VjkHJRVbKaqO+ul9lI11
XYkRQJ4G+h0fqV7qdcqLRzuqmnwUun7K3KdLdY7BuLuJbB947rtpZ9ZrcDMI9wY2
wEwJmstrO3CC9rK3pN1O64xFGxWzMFHC3xqsT1YW
-----END CERTIFICATE-----
/opt/Elastic/Agent/elastic-agent status
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a failed state
├─ beat/metrics-monitoring
│ ├─ status: (HEALTHY) Healthy: communicating with pid '10400'
│ ├─ beat/metrics-monitoring
│ │ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
│ └─ beat/metrics-monitoring-metrics-monitoring-beats
│ └─ status: (STARTING) Starting
├─ filestream-monitoring
│ ├─ status: (HEALTHY) Healthy: communicating with pid '10389'
│ ├─ filestream-monitoring
│ │ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
│ └─ filestream-monitoring-filestream-monitoring-agent
│ └─ status: (STARTING) Starting
└─ http/metrics-monitoring
├─ status: (HEALTHY) Healthy: communicating with pid '10410'
├─ http/metrics-monitoring
│ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
└─ http/metrics-monitoring-metrics-monitoring-agent
└─ status: (STARTING) Starting
Can the actual deployed (and used) versions of the files like beat-rendered-config.yml be directly manipulated?
I don't think it can be manipulated, as far as I know these configs are generated from the agent policy, so will be overwritten the next time the policy changes.
Hi Julia,
Based on my updates above do you have any other ideas or suggestions?
Also is the certificate it cannot read elasticsearch.ssl.certificate the self signed Elasticsearch actual certificate or the self signed CA certificate?
I see you have some questions above also. I will reply to them now.
Best Regards,
Kevin.
Hi Julia,
Answer to this question.
Do you have any fleet related config in kibana.yml file?
The only entry I can see is
"xpack.fleet.registryProxyUrl: http://vsdbahlprxy1:8080"
This is because the ELK servers are behind a proxy and without this Fleet cannot download from the Elastic image registry.
Below is full kibana.yaml
more kibana.yml
# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html
# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
#server.port: 8080
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"
server.host: "10.2.134.121"
# Enterprise Search instance
enterpriseSearch.host: 'http://localhost:3002'
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false
# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""
# the production wildcard DNS is *.apps.services.agriculture.gov.ie
# https://*.apps.services.agriculture.gov.ie/
#server.publicBaseUrl: "http://kibana-external.apps.services.agriculture.gov.ie/"
#server.publicBaseUrl: "http://rhoslog01.apps.services.agriculture.gov.ie/"
server.publicBaseUrl: "http://rhoslog01.agriculture.gov.ie/"
# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576
# The Kibana server's name. This is used for display purposes.
server.name: "kibana"
# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: false
#server.ssl.enabled: true
#server.ssl.certificate: /etc/kibana/kibana.crt
#server.ssl.key: /etc/kibana/kibana.key
# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://localhost:9200"]
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "XXXXXXXX"
elasticsearch.password: "XXXXXXXXXXXXXXXXX"
# Kibana can also authenticate to Elasticsearch via "service account tokens".
# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
# Use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# The maximum number of sockets that can be used for communications with elasticsearch.
# Defaults to `Infinity`.
#elasticsearch.maxSockets: 1024
# Specifies whether Kibana should use compression for communications with elasticsearch
# Defaults to `false`.
#elasticsearch.compression: false
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# =================== System: Elasticsearch (Optional) ===================
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# kibana is behind a proxy so need to tell it so it can download from elasticsearch
# From here https://discuss.elastic.co/t/error-plugins-fleet-failed-to-fetch-latest-version/335115 and
# here https://www.elastic.co/guide/en/kibana/current/fleet-settings-kb.html
xpack.fleet.registryProxyUrl: http://vsdbahlprxy1:8080
# Enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
elasticsearch.ssl.verificationMode: none
# default is full but cannot use for now as using self signed certificate
#elasticsearch.ssl.verificationMode: full
# =================== System: Logging ===================
# Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'info'
#logging.root.level: debug
# Enables you to specify a file where Kibana stores log output.
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
# layout:
# type: json
# Logs queries sent to Elasticsearch.
#logging.loggers:
# - name: elasticsearch.query
# level: debug
# Logs http responses.
#logging.loggers:
# - name: http.server.response
# level: debug
# Logs system usage information.
#logging.loggers:
# - name: metrics.ops
# level: debug
# =================== System: Other ===================
# The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data
#path.data: data
# Specifies the path where Kibana creates the process ID file.
pid.file: /run/kibana/kibana.pid
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000ms.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English (default) "en", Chinese "zh-CN", Japanese "ja-JP", French "fr-FR".
#i18n.locale: "en"
# =================== Frequently used (Optional)===================
# =================== Saved Objects: Migrations ===================
# Saved object migrations run at startup. If you run into migration-related issues, you might need to adjust these settings.
# The number of documents migrated at a time.
# If Kibana can't start up or upgrade due to an Elasticsearch `circuit_breaking_exception`,
# use a smaller batchSize value to reduce the memory pressure. Defaults to 1000 objects per batch.
#migrations.batchSize: 1000
# The maximum payload size for indexing batches of upgraded saved objects.
# To avoid migrations failing due to a 413 Request Entity Too Large response from Elasticsearch.
# This value should be lower than or equal to your Elasticsearch cluster’s `http.max_content_length`
# configuration option. Default: 100mb
#migrations.maxBatchSizeBytes: 100mb
# The number of times to retry temporary migration failures. Increase the setting
# if migrations fail frequently with a message such as `Unable to complete the [...] step after
# 15 attempts, terminating`. Defaults to 15
#migrations.retryAttempts: 15
# =================== Search Autocomplete ===================
# Time in milliseconds to wait for autocomplete suggestions from Elasticsearch.
# This value must be a whole number greater than zero. Defaults to 1000ms
#unifiedSearch.autocomplete.valueSuggestions.timeout: 1000
# Maximum number of documents loaded by each shard to generate autocomplete suggestions.
# This value must be a whole number greater than zero. Defaults to 100_000
#unifiedSearch.autocomplete.valueSuggestions.terminateAfter: 100000
[root@rhoslog01 kibana]#
Best Regards,
Kevin.
Hi Julia,
Also, do you have anything other than the certificate itself in this file --fleet-server-cert=/etc/elasticsearch/certs/fleet-server/fleet-server.crt ?
Answer to this question.
Not it is single certificate. See below with some redaction.
more /etc/elasticsearch/certs/fleet-server/fleet-server.crt
-----BEGIN CERTIFICATE-----
MIIDVTCCAj2gAwIBAgIURGvzoTRoxyxUjfJ9HZ9fk7HWMhwwDQYJKoZIhvcNAQEL
XXXXXX - Lots taken out
6wPJO4KHweGUgme3awNILzch6o77H0taYOqep95MovO1Am9X41ideoo=
-----END CERTIFICATE-----
A decode of the certificate is below.
{
"name": "\/CN=fleet-server",
"subject": {
"CN": "fleet-server"
},
"hash": "f0450f6b",
"issuer": {
"CN": "Elastic Certificate Tool Autogenerated CA"
},
"version": 2,
"serialNumber": "0x446BF3A13468C72C548DF27D1D9F5F93B1D6321C",
"serialNumberHex": "446BF3A13468C72C548DF27D1D9F5F93B1D6321C",
"validFrom": "240312145458Z",
"validTo": "270312145458Z",
"validFrom_time_t": 1710255298,
"validTo_time_t": 1804863298,
"signatureTypeSN": "RSA-SHA256",
"signatureTypeLN": "sha256WithRSAEncryption",
"signatureTypeNID": 668,
"purposes": {
"1": [
true,
false,
"sslclient"
],
"2": [
true,
false,
"sslserver"
],
"3": [
true,
false,
"nssslserver"
],
"4": [
true,
false,
"smimesign"
],
"5": [
true,
false,
"smimeencrypt"
],
"6": [
true,
false,
"crlsign"
],
"7": [
true,
true,
"any"
],
"8": [
true,
false,
"ocsphelper"
],
"9": [
false,
false,
"timestampsign"
]
},
"extensions": {
"subjectKeyIdentifier": "2D:06:29:4C:AE:F8:E2:88:C8:0C:EC:CB:88:A2:EB:EC:B3:53:A4:FB",
"authorityKeyIdentifier": "keyid:8D:BC:CE:07:AD:3A:AA:66:7E:9C:42:C0:66:3D:BC:76:F2:C8:5B:3B\n",
"subjectAltName": "DNS:rhoslog01.agriculture.gov.ie, IP Address:10.2.134.121",
"basicConstraints": "CA:FALSE"
}
}
Best Regards,
Kevin.
I don't really have any other ideas, I'm surprised setting the ssl.certificate in the advanced yaml config didn't help, it seems somewhere in the code the wrong value is still being preserved.
My suspicion is that the wrong certificate is the CA cert, because it is under the authorities: key, but I'm not sure where the problem is, the command arguments doesn't look wrong.
Hi Julia,
Thanks for your help so far on this problem.
I wonder could we set-up just fleet to not use TLS Certificates? Is it possible? Or does the fact that I have the Elasticsearch CA and Elasticsearch itself has self signed certificates mean fleet must use self signed certs? To be clear I have used the elasticsearch-certutil with same CA to generate the fleet server certificates.
Have you heard of other customers who are on premise and have self signed certificates (generated by elasticsearch-certutil) and fleet that are working?
Does each of elastic-agent -> beat/metrics-monitoring, filestream-monitoring, http/metrics-monitoring need an individual configuration with settings for certificates? Or should it use the settings from fleet itself?
Do any of your colleagues have any suggestions?
Best Regards,
Kevin.
You can just keep ssl between fleet-ES by keeping fleet-server-es-ca and remove the other certs. You will have to use insecure flags when installing fleet-server and agent as described here: Troubleshoot common problems | Fleet and Elastic Agent Guide [master] | Elastic
The agent components should use the the certs configured in fleet, not needed to configure separately.
I see that you pretty much followed the guide to configure security, so I'm sorry it doesn't seem to work. These options has been tested and working for a long time, so I'm not sure where the problem is.
I'll try to reproduce your issue locally.
Thanks Julia.
It would be great if you try to re-create. Summary of my environment is 3 x RHEL 8 VMs.
Node1 = Elastic+Kibana+Broken Fleet
Node2 = Elastic
Node3 = Elastic
I have used the elasticsearch certificate utiltiy to generate self signed CA and to generate all other certificates using the self signed CA.
I don't actually use a self signed certificate for the kibana GUI. It just uses http not https.
Just to say thanks for the apology but I know this is not "your fault" and you have been super helpful.
If you have any questions about my set-up please let me know. This might make it easier for you to replicate the problem.
Best Regards,
Kevin.
Hi Julia,
I tried this option in the advanced yaml options but it did not work.
ssl.certificate_authorities:
- |
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIVANEFkHTzX9WraDrfYzs/1/Thg9PYMA0GCSqGSIb3DQEB
STUFF CUT ...
wEwJmstrO3CC9rK3pN1O64xFGxWzMFHC3xqsT1YW
-----END CERTIFICATE-----
output.elasticsearch.ssl.certificate:
- |
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIVANEFkHTzX9WraDrfYzs/1/Thg9PYMA0GCSqGSIb3DQEB
STUFF CUT ...
wEwJmstrO3CC9rK3pN1O64xFGxWzMFHC3xqsT1YW
-----END CERTIFICATE-----
The fleet agent status is still the same.
/opt/Elastic/Agent/elastic-agent status
┌─ fleet
│ └─ status: (STARTING)
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a failed state
├─ beat/metrics-monitoring
│ ├─ status: (HEALTHY) Healthy: communicating with pid '1961'
│ ├─ beat/metrics-monitoring
│ │ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
│ └─ beat/metrics-monitoring-metrics-monitoring-beats
│ └─ status: (STARTING) Starting
├─ filestream-monitoring
│ ├─ status: (HEALTHY) Healthy: communicating with pid '1865'
│ ├─ filestream-monitoring
│ │ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
│ └─ filestream-monitoring-filestream-monitoring-agent
│ └─ status: (STARTING) Starting
├─ fleet-server-default
│ ├─ status: (HEALTHY) Healthy: communicating with pid '1855'
│ ├─ fleet-server-default
│ │ └─ status: (STARTING) waiting for input unit
│ └─ fleet-server-default-fleet-server-fleet_server-355fc92f-6a7b-444e-a10a-7d4027da7362
│ └─ status: (FAILED) can not convert 'object' into 'string' accessing 'output.elasticsearch.ssl.certificate'
└─ http/metrics-monitoring
├─ status: (HEALTHY) Healthy: communicating with pid '1994'
├─ http/metrics-monitoring
│ └─ status: (FAILED) could not start output: failed to reload output: could not setup output certificates reloader: unpacking 'ssl' config: can not convert 'object' into 'string' accessing 'elasticsearch.ssl.certificate'
└─ http/metrics-monitoring-metrics-monitoring-agent
└─ status: (STARTING) Starting
I also tried
output.elasticsearch.ssl.certificate: "/etc/elasticsearch/certs/elastic-stack-ca.crt"
But that did not work.
Best Regards,
Kevin.
What happens if you try to unenroll and re-enroll the agent? I'm wondering if the agent didn't get the new agent policy version.
Hi Julia,
I do a complete un-install and re-install each time so I think that would be covered. When I uninstall I get the uninstall token from the GUI. Then I remove the fleet agent completely from the GUI.
Then I generate new install token and use that in the original install command shared at the start of this discussion.
Best Regards,
Kevin.
Hi Julia,
I am just wondering did you get any time to try and reproduce the problem in your lab? Or do you have any further debugging suggestions?
The bigger question for me is how can I deploy a managed Kubernetes Agent on my Kubernetes Cluster with no Fleet Service?
Can this be done? Or should I wait until I can get the Fleet Service working?
Best Regards,
Kevin.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.