I have spun up an Elastic Cloud on Kubernetes cluster with Let's Encrypt SSL certificates with a domain name I own. All of these SSL certs are running properly, as well as my cluster. I have tested this locally and publicly.
When trying to install Elastic Agent on a Ubuntu 20.0.4 agent on my local network (happens to be the same subnet (10.10.2.1/24) as the ECK cluster) I am unable to get proper communication between the agent and the elastic cluster. When installing, I run:
After running the command, and getting a positive return message. The agent shows up in https://elastic.domain.example/app/fleet#/fleet/agents for a single "heartbeat" but does not ship any data (yes there is an assigned integration, linux). Upon looking into the log files, I noticed that it was trying to reach out to https://0.0.0.0:5601 (which since it is one of the nodes running my kubernetes stack is technically a valid way to reach kibana) however it errors out with a certificate error (expected) since 0.0.0.0 is not a valid domain under the valid cert.I dug into the fleet.yml file to look into the configuration of the agent and noticed that 0.0.0.0:5601 was listed under hosts:. I do not understand how this got there with the fresh installation listed, or why this agent is reaching out over that when I specified the Kibana url for it to use. Please let me if there is a way I can remove this, and force it to use the proper URL or if there is somehow a reason it is trying to do this. I have debugged this for a while with no luck. Thank you!
The errors in elastic-agent.log.1 are: 2021-02-12T21:08:31.363Z ERROR application/fleet_gateway.go:168 failed to dispatch actions, error: acknowledge 1 actions '[action_id: 35cb0910-6d72-11eb-83f4-45fadbc24bda, type: POLICY_CHANGE]' for elastic-agent '53d5a150-6d76-11eb-83f4-45fadbc24bda' failed: fail to ack to fleet: Post "https://0.0.0.0:5601/api/fleet/agents/53d5a150-6d76-11eb-83f4-45fadbc24bda/acks?": x509: cannot validate certificate for 0.0.0.0 because it doesn't contain any IP SANs
2021-02-12T21:29:40.458Z ERROR application/fleet_gateway.go:187 Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post "https://0.0.0.0:5601/api/fleet/agents/53d5a150-6d76-11eb-83f4-45fadbc24bda/checkin?": x509: cannot validate certificate for 0.0.0.0 because it doesn't contain any IP SANs
The fleet.yml contains: corey@k3s-node-1:$ sudo cat /opt/Elastic/Agent/fleet.yml
agent:
id: <not sure if this needs to be redacted, but it is>
fleet:
enabled: true
access_api_key: <redacted>
kibana:
protocol: https
host: elastic.domain.example:5601
hosts:
- 0.0.0.0:5601
timeout: 5m0s
reporting:
threshold: 10000
check_frequency_sec: 30
agent:
id: ""
Not sure if this was posted in the wrong spot. If it was please let me know and I will re-post. It would awesome to get some sort of support with this issue. I understand that the Elastic Agent is still in Beta, is it possible this is an actual bug with the install/enrollment script?
@coreysabia You need to set the correct URL for Kibana/elasticsearch on the Fleet page in Kibana.
If you go to Management > Fleet; click "Settings" in top right; modify the Kibana URL and elasticsearch URL under Global Output. This will update your policies and your Agent should start shipping data.
@blaker Thank you! Since I am moving from the hosted Elastic Cloud Service to ECK, this was a setting I had never touched before. Thank you for the quick and easy solution -- and keep up the awesome work on the platform!
@blaker It looks like I marked this as a solution a little too soon. Attempted to do exactly what you stated and am getting an error: Unable to load outputs. Below is the output of the error with the domain name changed.
This is now a non-issue and was related network ACL rules it was not related to the issue at hand and is now fixed. Remarking as solved with @blaker 's original solution which solved the original issue. Thank you again for the help.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.