Hello. I know this has been asked a million times here, but i've tried everything i know and i still can't get it to work.
I have an Agent Policies with Network Packet Capture, Elastic Defend, and System integrations.
When I add an agent to that Agent Policies, it’s healthy for 30 seconds and immediately goes Unhealthy.
System spesification
OS: Ubuntu 20.04 using VPS
Kernel version: 5.4.0 (more details on Output of /proc/version attached below)
Elasticsearch, kibana, elastic-agent version: 8.5.0
Elastic Defend integration version: 8.5.0
Network Packet Capture integration version: 1.7.0
System integration: 1.20.4
Command used to enroll agent:
./elastic-agent install --url=https://xxx.xxx.xxx.xxx:8220 --enrollment-token=[the-token] --insecure
Output of /proc/version
# cat /proc/version
Linux version 5.4.0 (mockbuild@builder9.eng.sw.ru) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Jan 25 12:49:12 MSK 2022
Output of elastic-agent status:
# elastic-agent status
Status: DEGRADED
Message: app endpoint-security--8.5.0-9d736430: Protecting with policy {5df063a9-fe73-400e-b276-4bca25262c8a}
Applications:
* endpoint-security (DEGRADED)
Protecting with policy {5df063a9-fe73-400e-b276-4bca25262c8a}
* filebeat (HEALTHY)
Running
* filebeat_monitoring (HEALTHY)
Running
* metricbeat (HEALTHY)
Running
* packetbeat (HEALTHY)
Running
Output of elastic-agent diagnostics:
# elastic-agent diagnostics
elastic-agent id: 0a63ea03-43db-49cd-b0e7-5b93a1f981aa version: 8.5.0
build_commit: 9da6ba5fce5d6b4d2c473c1f5ff6056794e9a644 build_time: 2022-10-24 20:21:40 +0000 UTC snapshot_build: false
Applications:
* name: endpoint-security route_key: default
error: Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory
Output of /opt/Elastic/Endpoint/state/log
# cat endpoint-000000.log | grep error
{"@timestamp":"2022-11-12T02:00:40.639363586Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":2876,"name":"Artifacts.cpp"}}},"message":"Artifacts.cpp:2876 Failed to download artifact diagnostic-configuration-v1 - Invalid url","process":{"pid":15896,"thread":{"id":15896}}}
{"@timestamp":"2022-11-12T02:00:40.639405139Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":647,"name":"Artifacts.cpp"}}},"message":"Artifacts.cpp:647 Artifact diagnostic-configuration-v1 download or verification failed","process":{"pid":15896,"thread":{"id":15896}}}
{"@timestamp":"2022-11-12T02:00:40.667775596Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":106,"name":"Internal.cpp"}}},"message":"Internal.cpp:106 sqlite3_prepare_v2 failed: rc=1, msg=SQL logic error","process":{"pid":15896,"thread":{"id":15896}}}
{"@timestamp":"2022-11-12T02:00:40.668425509Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":240,"name":"Tux_HostIsolation.cpp"}}},"message":"Tux_HostIsolation.cpp:240 Failed to mount bpf fs at /sys/fs/bpf: error 2","process":{"pid":15896,"thread":{"id":15908}}}
{"@timestamp":"2022-11-12T02:00:40.709841862Z","agent":{"id":"00000000-0000-0000-0000-000000000000","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":534,"name":"Comms.cpp"}}},"message":"Comms.cpp:534 No valid comms client configured","process":{"pid":15896,"thread":{"id":15896}}}
{"@timestamp":"2022-11-12T02:01:05.493419385Z","agent":{"id":"0a63ea03-43db-49cd-b0e7-5b93a1f981aa","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":2137,"name":"Config.cpp"}}},"message":"Config.cpp:2137 Initial configuration application failed","process":{"pid":15896,"thread":{"id":15944}}}
{"@timestamp":"2022-11-12T02:01:05.494140122Z","agent":{"id":"0a63ea03-43db-49cd-b0e7-5b93a1f981aa","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":288,"name":"AgentContext.cpp"}}},"message":"AgentContext.cpp:288 Failed to apply new policy from Agent.","process":{"pid":15896,"thread":{"id":15944}}}
{"@timestamp":"2022-11-12T02:01:05.688412681Z","agent":{"id":"0a63ea03-43db-49cd-b0e7-5b93a1f981aa","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self signed certificate in certificate chain]","process":{"pid":15896,"thread":{"id":15902}}}
{"@timestamp":"2022-11-12T02:05:47.523862196Z","agent":{"id":"0a63ea03-43db-49cd-b0e7-5b93a1f981aa","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":2144,"name":"Config.cpp"}}},"message":"Config.cpp:2144 Policy failed to apply and rollback is disabled","process":{"pid":15896,"thread":{"id":15905}}}
{"@timestamp":"2022-11-12T02:05:48.568076681Z","agent":{"id":"0a63ea03-43db-49cd-b0e7-5b93a1f981aa","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":327,"name":"Http.cpp"}}},"message":"Http.cpp:327 CURL error 60: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self signed certificate in certificate chain]","process":{"pid":15896,"thread":{"id":15902}}}
Security -> Endpoint page:
More detail:
What I have tried:
-
restarting and reinstalling elastic-agent
-
kernel security lockdown
I got the same error message as the post above "Failed to mount bpf fs at /sys".
The post above tells me to look into /sys/kernel/security/lockdown but, probably because i'm using VPS, I don't have that file. I tried making one and it doesn't let me even though i'm root.
# cat /sys/kernel/security/lockdown
cat: /sys/kernel/security/lockdown: No such file or directory
# vim lockdown
"lockdown" E212: Can't open file for writing
# nano lockdown
[ Error writing lockdown: Permission denied ]
I'm basically scratching my head at this point. Any help would be appreciated!!