I am currently running the latest version of Elastic Agent which is enrolled in the fleet but I don't see any endpoint enrolled yet. The message I get for a few days now in Security/Detections/Administration page is "Endpoints are enrolling. View agents to track progress". But I checked and agent is Online status now.
The steps that I followed up until now are:
installing elasticsearch cluster and kibana
securing communication between these 2 with a self signed certificate
installing elastic agent on a windows 10 machine following the official documentation
running the elastic agent in powershell
Can you please help me figure out the root cause of this issue?
If you need more info or screenshots please let me know.
Did you enable the endpoint security integration in Fleet? From what you wrote it seems you did but I'd like to verify.
Given that you are using a self signed certificate the issue may be that the root certificate is not installed on the host. Is it installed? If not, can you install it to see if that fixes this issue.
When you click on the Agent name in KIbana on the Fleet -> Agents page you will be taken to an activity log for that Agent. Are there any log messages in there related to endpoint-security that show errors?
Is Elastic Endpoint installed and running on the host? If installed it will be in c:\Program Files\Elastic\Endpoint. Logs are written to c:\Program Files\Elastic\Endpoint\state\log\endpoint-*.log. If you have logs flowing let me know and we can dig into them.
I am having the same problem. The root certificate is installed on the host and the intermediate is installed. In the endpoint security I get this in the log
Http.cpp:38 CURL error 60: Error [SSL certificate problem: unable to get local issuer certificate]
I checked the steps you mentioned and in the endpoint-*.log file I am receiving the following:
Elasticsearch connection is down",
HttpLib.cpp:1442 Establishing GET connection to [https://x.x.x.x:9200/_cluster/health]","process":{"pid":11780,"thread":{"id":9352}}}
"message":"Http.cpp:38 CURL error 60: Error [SSL certificate problem: unable to get local issuer certificate]
It seems that is the same error message that was provided by @AirJordan but I want to add something here: the CA certificate was generated with the elasticsearch-certutil command and followed the exact steps from official documentation. Furthermore the communication between elasticsearch and kibana is done properly over https as I can see the metrics logs in kibana (see below the screenshot)
That is the same for me also. Metrics are collecting fine. I added an IIS integration and most of that works on one of the servers. I understand it is a Beta Product so I don't expect everything to work but I can't get Endpoint Security to really work at all. I have https setup in kibana and elastic and kibana/elastic are talking fine to each other. I did have one client in working status but I stupidly thought the other ones weren't working for another reason and it removed it from the working one also. I will try and dig and see if I can find if I can make the other one work again and see if I can spot the difference.
I ended up getting mine fixed. For me I am doing this in a PoC but I have a root and intermediate cert. For some reason it also wanted the intermediate cert to be import into the root trusted store also, just not the intermediate. Very weird. Sorry I didn't do mine using the cert-util way so I don't know what it generates for a cert.
Can you confirm that you have the certificate authority generated by certutil installed on the host?
And that ElasticEndpoint has been restarted since doing so?
I was able to start up an endpoint with the same error message: Http.cpp:38 CURL error 60: Error [SSL certificate problem: unable to get local issuer certificate]
Then I was able to connect to elasticsearch by doing those two things. First, installing the self-signed certificate authority on the machine running ElasticEndpoint, and second, stopping and starting the ElasticEndpoint service. (Re-reading the system certificate authorities requires a new policy be applied, or ElasticEndpoint restart)
I installed certificate authority by doing the following on my Windows 10 box. Hopefully, the steps are similar for you:
Copy the generated self-signed elasticsearch certificate authority, ca.crt, to the endpoint host.
Double click ca.crt
Click Install Certificate
Select Local Machine
Click Next
Select Place all certificates in the following store
Click Browse
Select Trusted Root Certification Authorities
Click OK
Click Next
Click Finish
See a pop up The import was successful
Click OK to dismiss the pop up
Click OK to dismiss the Certficate window
I stopped and start the ElasticEndpoint service be doing the following:
First of all I want to thank all the people involved in solving this. Second of all I was able to fix this as per your reply. The part that was wrong from my side was that I selected Current user instead of Local machine.
Thanks for diagnosing this! To confirm, before you added the intermediate certificate Agent was able to connect to Fleet and appear active in the Fleet UI, it was just Endpoint Security that failed to send data to Elasticsearch?
Correct. I could see everything working including IIS integrations I added. It was just the Endpoint Security that was stuck. I was seeing it active in Fleet Management. The other issue I do not like is I can have a host with say Winlogbeats and then Elastic-Agent on it and it will show the client twice in the front page of Endpoint security. It will consider say poc.poc.org as one machine and then poc as another yet they are the same machine. Then if I specifically go under hosts and administration I only see it once. It was under the administration where they were not showing up.
Is there a chance that the CA root was installed a user by mistake and installing the intermediate certificate fixed the issue because it was installed for the entire system?
Is this question aimed at me? I had the root installed at the computer level, installed under trusted root. I had the intermediate installed at computer level installed under intermediate authority. The only way I could get it to work is at the computer level add the intermediate authority to the root folder also, then restart the service. All this was at the computer level not the user level.
I am experiencing the same result. I have 3 agents out on endpoints and they are showing as enrolled but there is no data in the metric-* index and the "Endpoints are enrolling" message persists. (default namespace)
I'm sorry you're having so much trouble. Would you be able to DM me the Endpoint logs in c:\Program Files\Elastic\Endoint\state\logs? I understand if you're uncomfortable sharing raw logs, I can follow up with log messages for you to look for, but full logs will help resolve this with less back and forth.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
