Breaking changes between elastic agent 8.16.1 and 8.16.3

Elastic Stack Elastic Agent
,
1

We were successfully running the elastic agent using the image 8.16.1 in a Kubernetes replica set with the following environment variables set:

FLEET_ENROLL = "1"
FLEET_URL="<my cloud instance> via https"
FLEET_ENROLLMENT_TOKEN="<redacted>"

The security department asked us to update to the latest image version(8.16.3) because the version 8.16.1 was known to have critical vulnerabilies.

I did so, and after just changing the minor image version I started getting the errors like:
Attempting to reconnect to backoff(elasticsearch( http:// elasticsearch: 9200)) with 4 reconnect attempt(s)"

I saw the documentation from elastic about the elastic agent environment variables: [Elastic Agent environment variables | Fleet and Elastic Agent Guide [8.17] and tried to update my variables to:

FLEET_SERVER_ENABLE = "1"
FLEET_SERVER_ELASTICSEARCH_HOST= "<my cloud instance> via https"
FLEET_SERVER_SERVICE_TOKEN="<redacted>"

But the issue still persists.
It looks like a breaking change between these minor versions,
please help!

2

Hi @mtr_sw,

Welcome! Can you confirm you are looking at the Elastic 8.16 documentation rather than the 8.17 documentation you state in your post? I don't see any breaking changes between minors 8.16.1 and 8.16.3 listed in the documentation, so I wouldn't expect you to need to change your variables.

Let us know!

3

FLEET_SERVER_ELASTICSEARCH_HOST (string) The Elasticsearch host for Fleet Server to communicate with. Overrides ELASTICSEARCH_HOST when set.

Default: http://elasticsearch:9200

What it looks like to me is that variable is not getting picked up.

This has been the setting in 8.16.1 it did not change between 8.16.1 and 8.16.3 so I am not sure how it was working before

Can you show more of the logs... can you show the actually lines in context...

Another source of that error can be if you have turned on monitoring in the policy and have not provide an output for the agent monitoring data.. .then it will default to http://elasticsearch:9200

4

Hi Carly,
thanks for pointing this out you are right the variables in the second code snippet were from the wrong elastic version.

5

Hi Stephen, thanks for the hint I tried to set this environment variable bus still the log with http://elasticsearch:9200 not reachable is there.
I have updated to the latest 8.17 image
In my desperate tries I have defined all possible variables:

ELASTICSEARCH_HOST=https://<instance>.fleet.privatelink.westeurope.azure.elastic-cloud.com:443
FLEET_SERVER_HOST=https://<instance>.fleet.privatelink.westeurope.azure.elastic-cloud.com:443
FLEET_SERVER_ELASTICSEARCH_HOST=https://<instance>.fleet.privatelink.westeurope.azure.elastic-cloud.com:443

But I am still receiving the error about the elasticsearch:9200:
Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): Get \"http://elasticsearch:9200\": lookup elasticsearch on 10.197.0.10:53: no such host

6

Hi @mtr_sw

I recommend running some of the diagnostic command lines

I would start with inspect

And please clarify, is this standalone mode or fleet managed? I'm not sure I'm clear on that

7

Hi Stephen,

we are using the elastic agent which is managed by the fleet.
when executing elastic-agent inspect command I get the output to be:

outputs:
  default:
    hosts: http://elasticsearch:9200
    password: <REDACTED>
    preset: balanced
    type: elasticsearch
    username: elastic

But I expected by specifying the Environment variables:

ELASTICSEARCH_HOST=https://<instance>.fleet.privatelink.westeurope.azure.elastic-cloud.com:443
FLEET_SERVER_HOST=https://<instance>.fleet.privatelink.westeurope.azure.elastic-cloud.com:443
FLEET_SERVER_ELASTICSEARCH_HOST=https://<instance>.fleet.privatelink.westeurope.azure.elastic-cloud.com:443

to point out to my cloud instance instead.
I also tried to do a manual enroll form the pod shell:
elastic-agent enroll https://<my cloud instance>.fleet.privatelink.westeurope.azure.elastic-cloud.com:443 --enrollment-token <redacted>
and I am getting the error:
Error: fail to enroll: fail to execute request to fleet-server: dial tcp [::1]:80: connect: connection refused

I want to mention that my agent resides behind a firewall which re-encrypts the SSL traffic with its own certificae, but the certificate is added inthe linux trust store.
A curl to the same location seems ok:

curl -v https://<my cloud instance>.fleet.privatelink.westeurope.azure.elastic-cloud.com:443
*   Trying <server ip>:443...
* TCP_NODELAY set
* Connected to <my cloud instance>.privatelink.westeurope.azure.elastic-cloud.com (<redacted>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
8

Check the default elasticseadch output

Fleet > settings

Screenshot 2025-02-19 at 11.35.01 AM
Screenshot 2025-02-19 at 11.35.01 AM3582×1358 313 KB

Then check the actual policy to make sure it is using the correct output

Screenshot 2025-02-19 at 11.37.22 AM
Screenshot 2025-02-19 at 11.37.22 AM3524×1570 418 KB

when you use enrollment those are what are used

9

Hi Stephen,
thanks for you answer, the settings in the Fleet do seem ok, Just want to mention that we have another 2 stages Q & P which use these settings successfully with the elastic agent version 8.16.1, but on D the newer image versions for the elastic image it does not work, it seems like the agent cannot reach the elastic cloud to pick up the settings from the elastic cloud.