Error-Failed version compatibility check with elasticsearch: tls: failed to verify certificate: x509: certificate signed by unknown authority

[Austin1 (Austin)]

Hi all ,
Please help me with setting up elastic agent container!
I got below error when I used podman run command followed:

Error-Failed version compatibility check with elasticsearch: tls: failed to verify certificate: x509: certificate signed by unknown authority

podman run --privileged -e FLEET_SERVER_ENABLE=1 -e FLEET_SERVER ELASTICSEARCH_HOST="https://xyz:9200" -e KIBANA_FLEET_HOST="http://xyz:5601/" -e KIBANA_FLEET_USERNAME="elastic" -e KIBANA FLEET PASSWORD="<password>" -e FLEET SERVER SERVICE TOKEN="**********IUQ==" -e FLEET_SERVER_POLICY_ID= *******-bb19e9cf" -e FLEET_SERVER_ES_CA= "/local/server/path/to/volumes/certs/http_ca.crt" -e INSECURE=true -p 8220:8220 imageID

My elasticsearch and kibana are working perfectly accessible at respectively where xyz is name of self hosted server name where I am working.

https://xyz:9200
http://xyz:5601

Below is my elasticsearch.yml

xpack.security.enabled: true
 
xpack.security.enrollment.enabled: true
 
xpack.security.http.ssl:
      enabled: true
      keystore.path: certs/http.p12
 
#Enable encryption and mutual authentication between cluster nodes
 
xpack.security.transport.ssl:
      enabled: true
      verification mode: certificate
      keystore.path: certs/transport.p12
      truststore.path: certs/transport.p12

Following are podman commands to start elasticsearch and kibana

podman run -e ES_JAVA_OPTS="-Xms4g-Xmx4g" --name kib01 --net host -p 5601:5601 imageID
podman run -e ES_JAVA_OPTS="-Xms4g-Xmx4g" --name kib01 --net host -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -it imageID

I am using 8.11.1 version images
For my elasticsearch security is auto enabled (Didn't create any custom certs)
Can someone please help me with this, I am stuck at this point from past one week.

[xeraa]

Is the -e FLEET_SERVER_ES_CA= "/local/server/path/to/volumes/certs/http_ca.crt" really available in the Fleet server? I'd double check in the logs what is happening there.

Alternatively, you could try --fleet-server-es-insecure when working with a self generated CA (see Elastic Agent command reference | Fleet and Elastic Agent Guide [8.11] | Elastic for the background of the command).

[Austin1 (Austin)]

Hi @xeraa
Thanks for your reply!
When I use --fleet-server-es-insecure flag
I got following:
Error: Request to get security token from kibana: Not found

[xeraa]

1. Kibana is up and running, right?
2. -e KIBANA FLEET PASSWORD looks like it's missing some underscores?

[Austin1 (Austin)]

Hi @xeraa
Thanks pointing command mistakes, while pasting the question I have missed those underscores but when I am running on my system underscores are present in the command. Also. my issue for certificate signed by unknown authority got resolved by using following command

podman run --privileged -e FLEET_SERVER_ENABLE=1 -e ELASTICSEARCH_HOST="https://xyz:9200" -e FLEET_SERVER ELASTICSEARCH_HOST="https://xyz:9200" -e KIBANA_FLEET_HOST="http://xyz:5601/" -e FLEET_SERVER_SERVICE_TOKEN="**********IUQ==" -e FLEET_SERVER_POLICY_ID= "test1" -e FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT = "********129966" -e FLEET_SERVER_HOST="xyz" -e FLEET_SERVER_HOST="8220" -p 8220:8220 -e FLEET_SERVER_INSECURE_HTTP="true" -e KIBANA_FLEET_SETUP='1' -e KIBANA_FLEET_HOST="https://xyz:5601" -e KIBANA_FLEET_USERNAME="elastic" --network=host imageID

Now I found a new issue under kibana logs I have following error messages
1.
Error[plugins.fleet]Error connecting to package registry: request to https://epr.elastic.co/search?kibana.version=8.11.1 failed, reason: connect ETIMEDOUT xx.xxx.xxx.xxx:xxx
2.
Error[plugins.fleet]Failed to fetch latest version of fleet_server from registry: Error connecting to package registry: request to https://epr.elastic.co/search?package=fleet_server&prerelease=false&kibana.version=8.11.1 failed reason: connect ETIMEDOUT xx.xxx.xxx.xxx:xxx

Also under Elastic agent logs I have following
"log.level":"info", "message": "waiting on policy with Fleet server integration":"test1", "component":{"binary":"fleet-server", "dataset":"elastic.agent.fleet_server", "id":"fleet-server-default", "type":"fleet-server", "log":{"source":"fleet-server-default"}, "service.type":"fleet-server", "state":"STARTING", "ecs.version":"1.6.0", "service.name":"fleet-server", "ecs.version":"1.6.0"}

I got the above message repeatedly and elasticagent stopped with following message
Fleet-server failed: context cancelled

My idea about this issue:
I am working on network traffic restricted environment, so I believe I am not able to access the mentioned URL
there are 2 proposed solutions to work in Air gapped environment

1. Use Proxy server
2. Hosting elastic package registry

I think both these solutions are not possible in my case.
Can you please help me resolving this or alternate solution?
Thank you so much for your patience and response!

[Austin1 (Austin)]

@xeraa please correct me if my understanding is wrong!

[xeraa]

You are right on the proxy or hosting your own (as documented in Air-gapped environments | Fleet and Elastic Agent Guide [8.11] | Elastic). I don't think there is any other way with Agent.

If that is absolutely impossible for your setup, what about falling back to Beats rather than using Fleet? Those shouldn't have that requirement.