Elastic Agent attempting to push data to the wrong URL

I have spun up an Elastic Cloud on Kubernetes cluster with Let's Encrypt SSL certificates with a domain name I own. All of these SSL certs are running properly, as well as my cluster. I have tested this locally and publicly.

When trying to install Elastic Agent on a Ubuntu 20.0.4 agent on my local network (happens to be the same subnet (10.10.2.1/24) as the ECK cluster) I am unable to get proper communication between the agent and the elastic cluster. When installing, I run:

corey@k3s-node-1:/elastic-agent-7.11.0-linux-x86_64$ sudo ./elastic-agent install -f --kibana-url=https://elastic.domain.example:5602 --enrollment-token=<redacted>

After running the command, and getting a positive return message. The agent shows up in https://elastic.domain.example/app/fleet#/fleet/agents for a single "heartbeat" but does not ship any data (yes there is an assigned integration, linux). Upon looking into the log files, I noticed that it was trying to reach out to https://0.0.0.0:5601 (which since it is one of the nodes running my kubernetes stack is technically a valid way to reach kibana) however it errors out with a certificate error (expected) since 0.0.0.0 is not a valid domain under the valid cert. I dug into the fleet.yml file to look into the configuration of the agent and noticed that 0.0.0.0:5601 was listed under hosts:. I do not understand how this got there with the fresh installation listed, or why this agent is reaching out over that when I specified the Kibana url for it to use. Please let me if there is a way I can remove this, and force it to use the proper URL or if there is somehow a reason it is trying to do this. I have debugged this for a while with no luck. Thank you!

The errors in elastic-agent.log.1 are:
2021-02-12T21:08:31.363Z ERROR application/fleet_gateway.go:168 failed to dispatch actions, error: acknowledge 1 actions '[action_id: 35cb0910-6d72-11eb-83f4-45fadbc24bda, type: POLICY_CHANGE]' for elastic-agent '53d5a150-6d76-11eb-83f4-45fadbc24bda' failed: fail to ack to fleet: Post "https://0.0.0.0:5601/api/fleet/agents/53d5a150-6d76-11eb-83f4-45fadbc24bda/acks?": x509: cannot validate certificate for 0.0.0.0 because it doesn't contain any IP SANs

2021-02-12T21:29:40.458Z ERROR application/fleet_gateway.go:187 Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post "https://0.0.0.0:5601/api/fleet/agents/53d5a150-6d76-11eb-83f4-45fadbc24bda/checkin?": x509: cannot validate certificate for 0.0.0.0 because it doesn't contain any IP SANs

The fleet.yml contains:
corey@k3s-node-1:$ sudo cat /opt/Elastic/Agent/fleet.yml

agent:
  id: <not sure if this needs to be redacted, but it is>
fleet:
  enabled: true
  access_api_key: <redacted>
  kibana:
    protocol: https
    host: elastic.domain.example:5601
    hosts:
    - 0.0.0.0:5601
    timeout: 5m0s
  reporting:
    threshold: 10000
    check_frequency_sec: 30
  agent:
    id: ""
1 Like

I’m also seeing this issue

1 Like

Not sure if this was posted in the wrong spot. If it was please let me know and I will re-post. It would awesome to get some sort of support with this issue. I understand that the Elastic Agent is still in Beta, is it possible this is an actual bug with the install/enrollment script?

Thanks for posting this issue!! @blaker Do you know what's going on here? TIA!!

1 Like

Hi Kaiyan, thank you for the reply!

If anyone looking at this issue needs any additional clarification please let me know.

@coreysabia You need to set the correct URL for Kibana/elasticsearch on the Fleet page in Kibana.

If you go to Management > Fleet; click "Settings" in top right; modify the Kibana URL and elasticsearch URL under Global Output. This will update your policies and your Agent should start shipping data.

1 Like

@blaker Thank you! Since I am moving from the hosted Elastic Cloud Service to ECK, this was a setting I had never touched before. Thank you for the quick and easy solution -- and keep up the awesome work on the platform!

@blaker It looks like I marked this as a solution a little too soon. Attempted to do exactly what you stated and am getting an error: Unable to load outputs. Below is the output of the error with the domain name changed.

_callee$@https://elastic.example.com:5601/37941/bundles/plugin/fleet/fleet.chunk.2.js:39:78843
l@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968491
s/o._invoke</<@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968245
_/</e[t]@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968848
settings_flyout_asyncGeneratorStep@https://elastic.example.com:5601/37941/bundles/plugin/fleet/fleet.chunk.2.js:39:74715
_next@https://elastic.example.com:5601/37941/bundles/plugin/fleet/fleet.chunk.2.js:39:75075
settings_flyout_asyncToGenerator/</<@https://elastic.example.com:5601/37941/bundles/plugin/fleet/fleet.chunk.2.js:39:75228
settings_flyout_asyncToGenerator/<@https://elastic.example.com:5601/37941/bundles/plugin/fleet/fleet.chunk.2.js:39:74953
onSubmit@https://elastic.example.com:5601/37941/bundles/plugin/fleet/fleet.chunk.2.js:39:79804
m@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:1026
S@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:1169
x/<@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:1315
x@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:1401
M@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:1850
O@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:1662
w@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:2011
Tn@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:29004
ce@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:114667
Mn@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:29620
Ln@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:30577
wn@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:29865
__kbnSharedDeps__</t.unstable_runWithPriority@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:383:3462
Hr@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:45442
se@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:114444
On@https://elastic.example.com:5601/37941/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375:29288

This is now a non-issue and was related network ACL rules it was not related to the issue at hand and is now fixed. Remarking as solved with @blaker 's original solution which solved the original issue. Thank you again for the help.