Elastic endpoint isn't installed after adding Elastic Defend integration

Hello world, I have a problem when testing Elastic Security with "Current plan:Basic".

OS Version: Ubuntu 20.04 for Elastic Agent and Elasticsearch-Kibana-Fleet serve
Elastic Stack Version: 8.13.4

I am using the superuser "elastic" to log in to Kibana.

I have an Elasticsearch-Kibana-Fleet server with an auto-signed certificate. I created a policy with some integrations, I downloaded the Elastic Agent on another host and enrolled the policy token. Everything seems alright until I try to add Elastic Defend to this policy.
The status of the machine is "Unhealthy", and the error message is "Elastic Defend requires Elastic Agent to be installed at the default installation path". I find that he package '/opt/Elastic/Endpoint' is not installed.

And the log is

11:00:49.668
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)
11:00:50.808
elastic_agent
[elastic_agent][error] Spawned new component endpoint-default: Elastic Defend requires Elastic Agent be installed at the default installation path

I put my conf file here maybe it can help:
-rw-rw---- 1 kibana kibana 7956 mai 10 13:13 kibana.yml:

server.port: 5601
server.host: "0.0.0.0"
server.publicBaseUrl: "https://kibana:5601"
server.ssl.enabled: true
server.ssl.certificateAuthorities: ["/etc/kibana/certs/ca/ca.crt"]
server.ssl.certificate: /etc/kibana/certs/kibana.crt
server.ssl.key: /etc/kibana/certs/kibana.key
elasticsearch.hosts: ["https://elastic:9200"]
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca/ca.crt" ]
elasticsearch.ssl.verificationMode: full
logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid

and:
-rwxrwxrwx 1 elasticsearch elasticsearch 4046 mai 10 11:06 elasticsearch.yml

cluster.name: es-demo
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: elastic
http.port: 9200
transport.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  certificate: certs/elastic/elastic.crt
  key: certs/elastic/elastic.key
  certificate_authorities: certs/ca/ca.crt
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["Elastic-Kibana-Fleet"]
http.host: 0.0.0.0

and the command to install elastic-agent is:

#curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.13.4-linux-x86_64.tar.gz
#tar xzvf elastic-agent-8.13.4-linux-x86_64.tar.gz
cd elastic-agent-8.13.4-linux-x86_64
sudo ./elastic-agent install --url=https://fleet-server:8220 --enrollment-token=My_Policy_Enrollment_token --fleet-server-es-ca=./ca-fleet.crt --certificate-authorities=./ca-fleet.crt

the message is:

Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
[ ===] Service Started  [3m50s] Elastic Agent successfully installed, starting enrollment.
[====] Waiting For Enroll...  [3m51s] {"log.level":"info","@timestamp":"2024-05-14T11:00:42.261+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://fleet-server:8220/","ecs.version":"1.6.0"}
[   =] Waiting For Enroll...  [3m52s] {"log.level":"info","@timestamp":"2024-05-14T11:00:43.537+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-05-14T11:00:43.543+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":300},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[   =] Done  [3m52s]                               
Elastic Agent has been successfully installed

I tried to uninstall the agent and reinstall it, but it didn't work. Do you have any ideas for this probleme? Did i miss some steps?

Best regards

I assume the second machine is also Ubuntu 22.04?

The installation message indicates that Agent was going to be installed in the default installation path /opt/Elastic/Agent. Is the /opt a regular directory on a root drive or a mount point?

Thank you for your reply, Lesio! :blush:

I have one Agent installed on Ubuntu 20 (and the second one on Ubuntu 16), and my server is also on Ubuntu 20. The directory /opt/ is a regular directory with the correct permissions (drwxr-xr-x). The path for all my Agents is /opt/Elastic/Agent/.

I also tested adding Elastic Defend to the policy of my fleet server, and it worked. I noticed that for the the 'Last updated by' is elastic, while for the (for my fleet server), the 'Last updated by' is system. I'm not sure if this has any impact.

Then, when I clicked to the page of the Data Collection policy (for my Agents), it detected that there are 2 agents which is correct. However, when I clicked the 'save' button, it said it would update 5 endpoints..."

I have no idea what happened to my Data Collection policy

Thanks a lot

The surprising number of endpoints, 5 instead of 2, might be caused by inactive endpoints which you can see going via Fleet page, adding it into view from the Status drop down

Btw, I think installation issues like yours might be also caused by se-linux configuration.

Btw 2, Ubuntu 16 is not supported by 8.13.x, see Support Matrix | Elastic

Hello lesio,
thanks for your reply
we do have a configuration that /opt for /home/opt for our project
do you think maybe that is the reason ?
Do you have an idea where can i redefine the path in elastic agent?

Thanks a lot

You have an agent installed in /home/opt instead of /opt? If this is the case, you cannot use the Elastic Defend integration on this agent as the endpoint integration does not support a custom path.

Hello Leandrojmp,

In ou case, /opt is a symlink to /home/opt, I didn't change the installation path:

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.13.4-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.13.4-linux-x86_64.tar.gz
cd elastic-agent-8.13.4-linux-x86_64

sudo ./elastic-agent install \
--url=https://fleet-server:8220 \
--enrollment-token=my_token
--fleet-server-es-ca=/path_ca/ca.crt \
--certificate-authorities=/path_ca/ca.crt

the installation is done and the fonction of agent have no problem, when i check the path is /opt/Elastic it looks like good...

SV-0001:/opt/Elastic# pwd
/opt/Elastic

i'm not sur if the problème is our symlink...

Agent supports custom base path and symlink, Endpoint (Elastic Defend) does not.

Yeah, but as mentioned the Defend integration does not support custom paths or symlinks.

It will not work until you install it directly on /opt, without using symlinks, or install it as a package using rpm or deb.

Try a bind mount:

sudo mount --bind /home/opt /mnt

This will need to be persistent across reboots though, so an entry to /etc/fstab will be necessary as well:

/home/opt /opt none defaults,bind 0 0
2 Likes

I tried to put /home/opt /opt none defaults,bind 0 0 in /etc/fstab, and then reboot my server, and then add Elastic Defend in my policy, it doesn't work with the same problem ;(

Can i find the elastic defend package in deb? may be it is a solution?

The Elastic Defend is an integration of Elastic Agent, it is not a separated tool, the main issue is that the Elastic Defend requires the agent to be installed on one of the default paths which are /opt/Elastic or /usr/share/elastic-agent.

But bind mounting /home/opt on /opt should work.

What errors do you have now?

in my case,
sudo mount --bind /home/opt /opt
doesn't work untile i tried to delete the symlink, unenroll the agent and re install the agent. There was away the same error, i think that isn't the best solution for me,

/usr/share/elastic-agent is also a default path to install elastic agent ?
which option that i need to modifier in this command?

sudo ./elastic-agent install \
--url=https://fleet-server:8220 \
--enrollment-token=my_token
--fleet-server-es-ca=/path_ca/ca.crt \
--certificate-authorities=/path_ca/ca.crt

To install in/usr/share/elastic-agent you need to install using rpm or deb.

If you install using .tar.gz the default path is /opt/Elastic, if you install using rpm or deb it will install under /usr/share/elastic-agent but will also have files in /etc/ and /var.

Check the installation layout for a better understand.

1 Like

Hello leandrojmp,
When i try to install the elastic-agent package in deb it works!!

I still have a question in this part:
here is my command to install the package:

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.13.4-amd64.deb
sudo dpkg -i elastic-agent-8.13.4-amd64.deb

sudo elastic-agent enroll \
--url=https://fleet-server:8220 \
--enrollment-token=my_policy_token \
--fleet-server-es-ca=/home/elastic-agent_8.13.4/ca.crt \
--certificate-authorities=/home/elastic-agent_8.13.4/ca.crt

sudo systemctl enable elastic-agent 
sudo systemctl start elastic-agent

And i receive this error :

Sélection du paquet elastic-agent précédemment désélectionné.
(Lecture de la base de données... 116876 fichiers et répertoires déjà installés.)
Préparation du dépaquetage de elastic-agent-8.13.4-amd64.deb ...
DĂ©paquetage de elastic-agent (8.13.4) ...
Paramétrage de elastic-agent (8.13.4) ...
create symlink /usr/share/elastic-agent/bin/elastic-agent to /var/lib/elastic-agent/data/elastic-agent-8.13.4-a2e31a/elastic-agent
Traitement des actions différées (« triggers ») pour systemd (245.4-4ubuntu3.17) ...
{"log.level":"info","@timestamp":"2024-06-10T15:52:40.121+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://fleet-server:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-10T15:52:40.956+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-06-10T15:52:40.958+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 2s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-10T15:52:42.958+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 1","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-06-10T15:52:42.959+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 4s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-10T15:52:46.960+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 2","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-06-10T15:52:46.961+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 2 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 8s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-10T15:52:54.963+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 3","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-06-10T15:52:54.964+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 3 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 16s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-10T15:53:10.965+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 4","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-06-10T15:53:10.966+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Restart attempt 4 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 32s","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-06-10T15:53:42.967+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":296},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory\"","ecs.version":"1.6.0"}
Something went wrong while enrolling the Elastic Agent: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory"
Error: could not reload agent daemon, unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory"
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.13/fleet-troubleshooting.html
Synchronizing state of elastic-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable elastic-agent
Created symlink /etc/systemd/system/multi-user.target.wants/elastic-agent.service → /lib/systemd/system/elastic-agent.service.

My Elastic Agent is up and present in the fleet-server, the /var/lib/elastic-agent/elastic-agent.sock= file is created, but it's not /var/lib/elastic-agent/elastic-agent.sock, it has an = at the end.

Do you have any idea of this error?

Yeah, this is a know issue: [Fleet]: On enrolling RPM and Deb agents, `Restarting agent failed` error is displayed in CLI. · Issue #4084 · elastic/elastic-agent · GitHub

When enrolling the agent it first tries to stop the agent, but if there is no agent running it will give you this error.

As an alternative you can start the service before using the enroll command or use the --dealy-enroll parameter in the enroll command.

It works,thank your very much :slight_smile: