Unable to run Endpoint 8.4 on a Ubuntu 20.04 host hardened with CIS Level 2

dapster · November 9, 2022, 3:46pm

Hi,

I am encountering difficulties to properly run Endpoint (deployed thanks to Fleet Server) on a Ubuntu 20.04 host (hardened with CIS Level 2 and I suspect the issue to be related to system hardening).

When I run elastic agent diagnostic I have:

elastic-agent  id: bdf61569-dedf-4905-a8a9-207ef1391e8a                version: 8.4.2
               build_commit: d3eb3e8a02d96bf48d11145fb089e118f75bd202  build_time: 2022-09-14 22:33:40 +0000 UTC  snapshot_build: false
Applications:
  *  name: metricbeat             route_key: default
     process: metricbeat          id: ccd56c6c-f046-47be-894a-b6945206436f          ephemeral_id: 2a684434-eefc-4fc2-927f-55c118501809  elastic_license: true
     version: 8.4.2               commit: b00a6bca7be493b01a134a6ad8c415f2be297414  build_time: 2022-09-13 21:56:36 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS      username: root                                    user_id: 0                                          user_gid: 0
  *  name: packetbeat             route_key: default
     process: packetbeat          id: a73fec1b-61e0-4c20-ba92-f606c047be39          ephemeral_id: 0195e653-cfbf-4c84-870f-ecafcb333960  elastic_license: true
     version: 8.4.2               commit: b00a6bca7be493b01a134a6ad8c415f2be297414  build_time: 2022-09-13 21:52:18 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS      username: root                                    user_id: 0                                          user_gid: 0
  *  name: filebeat_monitoring    route_key: default
     process: filebeat            id: 6673dbeb-3cf5-4dca-bd01-25bbd36dc739          ephemeral_id: c986fc2d-3fa1-4a85-974c-ce7f96bfef47  elastic_license: true
     version: 8.4.2               commit: b00a6bca7be493b01a134a6ad8c415f2be297414  build_time: 2022-09-13 21:52:16 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS      username: root                                    user_id: 0                                          user_gid: 0
  *  name: metricbeat_monitoring  route_key: default
     process: metricbeat          id: ccd56c6c-f046-47be-894a-b6945206436f          ephemeral_id: 2a684434-eefc-4fc2-927f-55c118501809  elastic_license: true
     version: 8.4.2               commit: b00a6bca7be493b01a134a6ad8c415f2be297414  build_time: 2022-09-13 21:56:36 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS      username: root                                    user_id: 0                                          user_gid: 0
  *  name: endpoint-security      route_key: default
     error: Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory
  *  name: filebeat           route_key: default
     process: filebeat        id: 6673dbeb-3cf5-4dca-bd01-25bbd36dc739          ephemeral_id: c986fc2d-3fa1-4a85-974c-ce7f96bfef47  elastic_license: true
     version: 8.4.2           commit: b00a6bca7be493b01a134a6ad8c415f2be297414  build_time: 2022-09-13 21:52:16 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS  username: root                                    user_id: 0                                          user_gid: 0

As you can see, Endpoint encounters a Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory issue. Initially, the endpoint-security folder was not exist at all but I have the same result after creating it manually and setting a generous 755 chmod on the full path towards this brand new folder (this is a typical side effect of having a restrictive UMASK due to CIS hardening) .

If I run Endpoint output test I have:

Testing output connections using config file: [/opt/Elastic/Endpoint/elastic-endpoint.yaml]

Using proxy:

Elasticsearch server: http://ELK-1:9200
	Status: Success

Elasticsearch server: http://ELK-2:9200
	Status: Success

Global artifact server: https://artifacts.security.elastic.co
	Status: Success

Fleet server: https://ELK-1:8220
	Status: Success

Fleet server: https://ELK-2:8220
	Status: Success

If I look in the Endpoint logs looking for errors I am getting:

{"@timestamp":"2022-11-09T13:39:16.811850194Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":123,"name":"AgentConnectionInfo.cpp"}}},"message":"AgentConnectionInfo.cpp:123 Agent process is not root/admin or validation failed, disconnecting","process":{"pid":812513,"thread":{"id":812547}}}
{"@timestamp":"2022-11-09T13:39:16.811872511Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":588,"name":"AgentComms.cpp"}}},"message":"AgentComms.cpp:588 Unable to retrieve connection info from Agent(Agent is not running as root)","process":{"pid":812513,"thread":{"id":812547}}}
{"@timestamp":"2022-11-09T13:41:51.763655857Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":918,"name":"Network.cpp"}}},"message":"Network.cpp:918 Failed to stat [/proc/813679/fd/19]","process":{"pid":813825,"thread":{"id":813825}}}

Quite strange to see a Agent process is not root/admin or validation failed issue since the diagnostics confirm elastic agent is running as root and this is the same when I check the various processes with a ps -ef | grep Endpoint or ps -ef | grep elastic-agent.

The failed to stat issue points towards a link with the following rights:

lrwx------ 1 root root 64 Nov  9 15:27 /proc/813679/fd/19 -> 'socket:[472216859]'

Strangely, these errors do not block totally Endpoint since I can see some logs stating that documents are sent to elastic:

{"@timestamp":"2022-11-09T15:34:46.168982189Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":199,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:199 Sent 124 documents to Elasticsearch","process":{"pid":814313,"thread":{"id":814336}}}

In fact, I am investigating this issue because I have noticed regular crashs of the Endpoint process leading to create huge codedump files in /var/lib/apport/coredump (with a FS completely full...).

Important : please note I have legacy beats installed and running on the same host on top of the fleet elastic agent (we are migrating from Elastic 6.1 towards 8.4 and are testing Feet + Security Endpoint for now and will completely remove the legacy Beat at a later stage).

Any help will be highly appreciated !
Thanks

ferullo · November 14, 2022, 2:01pm

HI @dapster

Do you continue to have the issue you showed from running elastic-agent diagnostics , or did it go away after you created the endpoint-security directory?

The issue you found in the /opt/Elastic/Endpoint/state/log/ files looks like a transient issue that resolved itself. That error means that Endpoint could not connect to Agent to receive configuration updates. However, your elastic-endpoint test output command output shows that Endpoint has received configuration info, since it knows how to reach Elasticsearch and Fleet server. Endpoint being able to write to Elasticsearch also confirms this.

I wonder if Endpoint is crashing and causing the UNHEALTHY status in Fleet? What is the status of the Endpoint in the Security App's status page in Kibana? Regardless of whether or not Endpoint is crashing, it would be good to understand whether or not before crashing Endpoint is healthy. Can you share the output from elastic-agent status? That will help verify why Agent remains UNHEALTHY.

As an aside, Endpoint logs can also show the status by grepping for Policy action in the logs. Each time Endpoint applies policy it will produce a bunch of log messages for each action it takes while applying policy. Any that fail will have a failure log messages as well as a failure displayed in Kibana.

dapster · November 14, 2022, 5:26pm

Hi Daniel,

Thanks for getting back to me.

endpoint-security as located in a tmp folder does not persists over elastic-agent restart. I have just recreated it and restart ElasticEndpoint.service. The diagnostics issue is still there (and the /endpoint-security.sock file is still missing).

Regarding elastic-agent status, I am getting

Status: HEALTHY
Message: (no message)
Applications:
  * endpoint-security      (HEALTHY)
                           Protecting with policy {073b2f96-41a6-40c0-a943-c3fc7932ce18}
  * packetbeat             (HEALTHY)
                           Running
  * filebeat               (HEALTHY)
                           Running
  * metricbeat             (HEALTHY)
                           Running
  * filebeat_monitoring    (HEALTHY)
                           Running
  * metricbeat_monitoring  (HEALTHY)
                           Running

The status in Security/Manage/Endpoints UI is Healthy as well.

Grepping Policy action give the following :

{"@timestamp":"2022-11-14T17:00:30.436409883Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_memory_threat: success - Succesfully read memory protection configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.436430004Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_memory_threat: success - Succesfully read diagnostic memory protection configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.436508208Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_host_isolation: success - Successfully read host isolation configuration (disabled)","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.436538116Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malicious_behavior: success - Successfully read behavior protection configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.436553867Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malicious_behavior: success - Successfully read diagnostic behavior protection configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.436708428Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_user_notification: success - Succesfully read user notification configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.436734593Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malware: success - Successfully read malware prevent configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.43679971Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malware: success - Successfully read diagnosic malware off configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.444572162Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_output: success - Successfully read output configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.44484463Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_logging: success - Successfully read logging configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.444859699Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action load_config: success - Successfully parsed configuration","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.446535232Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action download_user_artifacts: success - Successfully downloaded user artifacts","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.459928625Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action download_global_artifacts: success - Global artifacts are available for use","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.460061929Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_logging: success - Successfully configured logging","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:30.461400125Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_output: success - Successfully configured output connection","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.044516422Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action detect_process_events: success - Success enabling process events; current state is enabled Source configuration changed.","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.044564259Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action detect_network_events: success - Success enabling network events; current state is enabled Source configuration changed.","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.044591399Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action detect_file_write_events: success - Success enabling file events; current state is enabled Source configuration changed.","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.044616992Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_file_events: success - Success enabling file events; current state is enabled Source configuration changed.","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.044636777Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_network_events: success - Success enabling network events; current state is enabled Source configuration changed.","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.044657035Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_process_events: success - Success enabling process events; current state is enabled Source configuration changed.","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.049203851Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malware: success - Successfully loaded malware model","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.049249883Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malware: success - Malware detection/prevention is disabled","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.049998151Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malware: success - Malware detection/prevention is disabled","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.082296525Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malware: success - Successfully loaded malware model","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.083111279Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malware: success - Successfully enabled malware prevention","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.083131029Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malware: success - Successfully disabled malware protection","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.08316764Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_user_notification: success - Successfully configured user notification","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.084543742Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malicious_behavior: success - Rules engine is stopped","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.084565793Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malicious_behavior: success - Rules engine is stopped","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.084753364Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_host_isolation: success - Activated exception-list IPs for host isolation","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.08477378Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_host_isolation: success - Host isolation exception list (processes) set","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.08509106Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_host_isolation: success - Host is not isolated","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.085130479Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_memory_threat: success - Memory scan is disabled","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.085148896Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_memory_threat: success - Memory scan is disabled","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.085167793Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_memory_threat: success - Successfully disabled memory threat protection","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:42.085185909Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_memory_threat: success - Successfully disabled memory threat protection","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:47.361510306Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_memory_threat: success - Succesfully read memory protection configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.36152993Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_memory_threat: success - Succesfully read diagnostic memory protection configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.361605971Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_host_isolation: success - Successfully read host isolation configuration (disabled)","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.361635618Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malicious_behavior: success - Successfully read behavior protection configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.361651827Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malicious_behavior: success - Successfully read diagnostic behavior protection configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.3618046Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_user_notification: success - Succesfully read user notification configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.36183047Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malware: success - Successfully read malware prevent configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.361895469Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malware: success - Successfully read diagnosic malware off configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.370931771Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_output: success - Successfully read output configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.371200469Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_logging: success - Successfully read logging configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:47.371215187Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action load_config: success - Successfully parsed configuration","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.086095036Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action agent_connectivity: success - Successfully connected to Agent","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:48.086539962Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action workflow: success - Successfully executed all workflows","process":{"pid":16672,"thread":{"id":16672}}}
{"@timestamp":"2022-11-14T17:00:48.090082183Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action download_user_artifacts: success - Successfully downloaded user artifacts","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.105119769Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action download_global_artifacts: success - Global artifacts are available for use","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.105280812Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_logging: success - Successfully configured logging","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.106191042Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_output: success - Successfully configured output connection","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.106257906Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action detect_process_events: success - Success enabling process events; current state is enabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.106275329Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action detect_network_events: success - Success enabling network events; current state is enabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.106292239Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action detect_file_write_events: success - Success enabling file events; current state is enabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.106309786Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_file_events: success - Success enabling file events; current state is enabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.106326769Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_network_events: success - Success enabling network events; current state is enabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.106345372Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_process_events: success - Success enabling process events; current state is enabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.110191263Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malware: success - Successfully loaded malware model","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.110214089Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malware: success - Malware detection/prevention is disabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.110884641Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malware: success - Malware detection/prevention is disabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.144573688Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malware: success - Successfully loaded malware model","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.144752063Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malware: success - Successfully enabled malware prevention","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.144769841Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malware: success - Successfully disabled malware protection","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.144813415Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_user_notification: success - Successfully configured user notification","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.145791563Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_malicious_behavior: success - Rules engine is stopped","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.145813546Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_malicious_behavior: success - Rules engine is stopped","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.145957702Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_host_isolation: success - Activated exception-list IPs for host isolation","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.145977165Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_host_isolation: success - Host isolation exception list (processes) set","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.146246727Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_host_isolation: success - Host is not isolated","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.14628559Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_memory_threat: success - Memory scan is disabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.146303688Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_memory_threat: success - Memory scan is disabled","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.146322884Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_memory_threat: success - Successfully disabled memory threat protection","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.146341621Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action configure_diagnostic_memory_threat: success - Successfully disabled memory threat protection","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.14638159Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action agent_connectivity: success - Successfully connected to Agent","process":{"pid":16672,"thread":{"id":16739}}}
{"@timestamp":"2022-11-14T17:00:48.146537325Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":357,"name":"Response.cpp"}}},"message":"Response.cpp:357 Policy action workflow: success - Successfully executed all workflows","process":{"pid":16672,"thread":{"id":16739}}}

I have stopped legacy beats to avoid any potential conflicts but I am still having coredumps in /var/lib/apport/coredump, last one created at 16:46UTC today. Corresponding security endpoint logs are (short extract, I can provide more if needed):

{"@timestamp":"2022-11-14T16:46:21.838967777Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":47,"na
me":"MainPosix.cpp"}}},"message":"MainPosix.cpp:47 Processed core-ing signal : 11","process":{"pid":15770,"thread":{"id":15798}}}
{"@timestamp":"2022-11-14T16:46:40.155316887Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":131,"name":"Config.cpp"}}},"message":"Conf
ig.cpp:131 Registered configuration callback for logging","process":{"pid":16257,"thread":{"id":16257}}}
{"@timestamp":"2022-11-14T16:46:40.155362953Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":155,"name":"Entry.cpp"}}},"message":"Entry
.cpp:155 Loading plugin: documentLogging","process":{"pid":16257,"thread":{"id":16257}}}
{"@timestamp":"2022-11-14T16:46:40.156732332Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":131,"name":"System.cpp"}}},"message":"Syst
em.cpp:131 Generated system UUID 99bc2f02-e951-4b1f-3ae6-33c79dccb4ba","process":{"pid":16257,"thread":{"id":16257}}}
{"@timestamp":"2022-11-14T16:46:40.156811274Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":804,"name":"LogStorage.cpp"}}},"message":"
LogStorage.cpp:804 Calculated outValue 1668441248 for filepath /opt/Elastic/Endpoint/state/documents/documents-2022-11-14T155408.log","process":{"pid":16257,"thread":{"id":16257}}}
{"@timestamp":"2022-11-14T16:46:40.210450417Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":131,"name":"System.cpp"}}},"message":"Syst
em.cpp:131 Generated system UUID 99bc2f02-e951-4b1f-3ae6-33c79dccb4ba","process":{"pid":16257,"thread":{"id":16257}}}
{"@timestamp":"2022-11-14T16:46:40.210474074Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":155,"name":"Entry.cpp"}}},"message":"Entry
.cpp:155 Loading plugin: comms","process":{"pid":16257,"thread":{"id":16257}}}
{"@timestamp":"2022-11-14T16:46:40.210532047Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":131,"name":"Config.cpp"}}},"message":"Conf
ig.cpp:131 Registered configuration callback for comms","process":{"pid":16257,"thread":{"id":16257}}}

Available to provide more insights, thanks a lot.

wsouza · November 14, 2022, 7:31pm

Isn't it missing some binary compilation library? Is the build-essential package installed? Try removing the old version of beats and updating your operating system and then reinstalling the elastic agent.

dapster · November 15, 2022, 7:25am

gcc --version responds correctly, so for me build-essential is correctly installed.

When looking to any warning/error following a fresh restart of Endpoint service, I can see only one warning related to Tux_HostIsolation and stating there is an issue with a missing file and a communication issue with the Kernel (probably the endpoint-security.sock mentioned in the diagnostics).

{"@timestamp":"2022-11-15T06:47:43.713087807Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":374,"name":"Tux_HostIsolation.cpp"}}},"mes
sage":"Tux_HostIsolation.cpp:374 Load and attach worked with load method=0","process":{"pid":18933,"thread":{"id":18959}}}
{"@timestamp":"2022-11-15T06:47:43.754937294Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":133,"name":"Tux_HostIsolation.cpp"}}},"mes
sage":"Tux_HostIsolation.cpp:133 IsEbpfSupported returned [1]","process":{"pid":18933,"thread":{"id":18959}}}
{"@timestamp":"2022-11-15T06:47:43.767214245Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":374,"name":"Tux_HostIsolation.cpp"}}},"mes
sage":"Tux_HostIsolation.cpp:374 Load and attach worked with load method=0","process":{"pid":18933,"thread":{"id":18933}}}
{"@timestamp":"2022-11-15T06:47:43.798925945Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"file":{"line":133,"name":"Tux_HostIsolation.cpp"}}},"mes
sage":"Tux_HostIsolation.cpp:133 IsEbpfSupported returned [1]","process":{"pid":18933,"thread":{"id":18933}}}
{"@timestamp":"2022-11-15T06:48:01.40459332Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"warning","origin":{"file":{"line":84,"
name":"Tux_HostIsolation.cpp"}}},"message":"rtnetlink replied: No such file or directory\n","process":{"pid":18933,"thread":{"id":18933}}}
{"@timestamp":"2022-11-15T06:48:01.404610059Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"warning","origin":{"file":{"line":84,
"name":"Tux_HostIsolation.cpp"}}},"message":"error talking to the kernel (rtnetlink_send)\n","process":{"pid":18933,"thread":{"id":18933}}}
{"@timestamp":"2022-11-15T06:48:01.470162859Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"warning","origin":{"file":{"line":84,
"name":"Tux_HostIsolation.cpp"}}},"message":"rtnetlink replied: No such file or directory\n","process":{"pid":18933,"thread":{"id":18981}}}
{"@timestamp":"2022-11-15T06:48:01.470180136Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"warning","origin":{"file":{"line":84,
"name":"Tux_HostIsolation.cpp"}}},"message":"error talking to the kernel (rtnetlink_send)\n","process":{"pid":18933,"thread":{"id":18981}}}

wsouza · November 15, 2022, 11:43am

On this host, is SELINUX active or some other similar solution? If possible, create a new VM, with the same operating system, however, without hardening it to better evaluate it, considering that the hardening applied may be interfering with this communication.

Removed all beats installed on the machine and tried to reinstall Elastic Agent?

joean407 · November 16, 2022, 9:10pm

Hello.

At the current time we are also seeing issues with ElasticAgent and Ubuntu 20.04 at CIS level 2. Our issues manifest themselves as the ElasticAgent and Elastic Endpoint Security showing up as healthy, but once a malware test is carried out on the host Elastic Endpoint Security doesn't stop anything.

We currently have a case with Elastic on this and are investigating, I just wanted to hint that there are other users seeing similar issues, and I will post any findings we discover.

Cheers
//Joel

Nick_Berlin · November 28, 2022, 8:08pm

I'm curious, what is the output of:
$ cat /proc/sys/kernel/ftrace_enabled

joean407 · November 29, 2022, 12:19pm

Short update to the issues we were seeing, as far as we can tell at the current point they are all related to having /tmp on a separate partition and tmpfs.

The original issue was pinned down to be that when trying malware test files in the /tmp directory Endpoint Security doesn't prevent these malware files to be copied, edited, et.c. as expected. However this only seems to apply to the /tmp directory and the case when /tmp is on a tmpfs filesystem, our case on this is ongoing.

Unfortunately I think this is unrelated to the issues seen in this case, sorry about that.

dapster · November 29, 2022, 1:34pm

Hi, thanks for the update and your analysis.

I have provided insights to Elastic team and they have identified a fix to be delivered in 8.5.3 version to released soon.

We are planning a migration from 8.4 to 8.5 in order to confirm the fix is working properly.

dapster · December 9, 2022, 2:03pm

After migration towards Elastic 8.5.3 I have checked the status of Endpoint Security.

elastic-agent diagnostics results are showing an issue with Security endpoint:

elastic-agent  id: bdf61569-dedf-4905-a8a9-207ef1391e8a                version: 8.5.3
               build_commit: 0e1a7396f356f8af5ad28c3c8e941256f50c86a2  build_time: 2022-12-06 00:04:15 +0000 UTC  snapshot_build: false
Applications:
  *  name: filebeat               route_key: default
     process: filebeat            id: 6673dbeb-3cf5-4dca-bd01-25bbd36dc739          ephemeral_id: 37165460-87a3-44a3-8117-c3926c880a02  elastic_license: true
     version: 8.5.3               commit: 6d03209df870c63ef9d59d609268c11dfdc835dd  build_time: 2022-12-04 04:51:48 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS      username: root                                    user_id: 0                                          user_gid: 0
  *  name: metricbeat             route_key: default
     process: metricbeat          id: ccd56c6c-f046-47be-894a-b6945206436f          ephemeral_id: a3127b07-103b-4548-84ef-ef9e1a1ee6e0  elastic_license: true
     version: 8.5.3               commit: 6d03209df870c63ef9d59d609268c11dfdc835dd  build_time: 2022-12-04 04:55:28 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS      username: root                                    user_id: 0                                          user_gid: 0
  *  name: filebeat_monitoring    route_key: default
     process: filebeat            id: 645ef588-221c-431e-b5c7-38b7f1b852a9          ephemeral_id: 94207ff3-e4d6-4c79-8e4a-afbc559687d8  elastic_license: true
     version: 8.5.3               commit: 6d03209df870c63ef9d59d609268c11dfdc835dd  build_time: 2022-12-04 04:51:48 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS      username: root                                    user_id: 0                                          user_gid: 0
  *  name: metricbeat_monitoring  route_key: default
     process: metricbeat          id: 62349d59-dc4b-49bb-9113-9ded59e81de9          ephemeral_id: 9c4bc512-0198-40b3-9228-a8ad32897a26  elastic_license: true
     version: 8.5.3               commit: 6d03209df870c63ef9d59d609268c11dfdc835dd  build_time: 2022-12-04 04:55:28 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS      username: root                                    user_id: 0                                          user_gid: 0
  *  name: packetbeat             route_key: default
     process: packetbeat          id: a73fec1b-61e0-4c20-ba92-f606c047be39          ephemeral_id: ab5931b8-6de8-459b-a2f2-417dd65f6471  elastic_license: true
     version: 8.5.3               commit: 6d03209df870c63ef9d59d609268c11dfdc835dd  build_time: 2022-12-04 04:54:34 +0000 UTC           binary_arch: amd64
     hostname: STAGING-LB-AS      username: root                                    user_id: 0                                          user_gid: 0
  *  name: endpoint-security      route_key: default
     error: Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory

Looking for errors in Endpoint logs:

{"line":123,"name":"AgentConnectionInfo.cpp"}}},"message":"AgentConnectionInfo.cpp:123 Agent process is not root/admin or validation failed, disconnecting","process":{"pid":1118,"thread":{"id":1381}}}
{"@timestamp":"2022-12-09T07:19:50.366625674Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":587,"name":"AgentComms.cpp"}}},"message":"AgentComms.cpp:587 Unable to retrieve connection info from Agent(Agent is not running as root)","process":{"pid":1118,"thread":{"id":1381}}}
{"@timestamp":"2022-12-09T13:15:44.004835589Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":2880,"name":"Artifacts.cpp"}}},"message":"Artifacts.cpp:2880 Failed to download artifact diagnostic-configuration-v1 - Invalid url","process":{"pid":3661,"thread":{"id":3661}}}
{"@timestamp":"2022-12-09T13:15:44.004863193Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":647,"name":"Artifacts.cpp"}}},"message":"Artifacts.cpp:647 Artifact diagnostic-configuration-v1 download or verification failed","process":{"pid":3661,"thread":{"id":3661}}}
{"@timestamp":"2022-12-09T13:15:44.142448699Z","agent":{"id":"00000000-0000-0000-0000-000000000000","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":534,"name":"Comms.cpp"}}},"message":"Comms.cpp:534 No valid comms client configured","process":{"pid":3661,"thread":{"id":3661}}}

Apparently, there is still an issue with:

"AgentConnectionInfo.cpp:123 Agent process is not root/admin or validation failed, disconnecting"

Agent is launched as root:

root        3661       1  2 13:15 ?        00:01:01 /opt/Elastic/Endpoint/elastic-endpoint run
root        1133       1  0 07:19 ?        00:01:14 elastic-agent

I am afraid the issue remains and will confirm later if the core dumps still appears.

Thanks

ferullo · December 10, 2022, 7:38pm

The process you confirmed is running as root is Elastic Endpoint, not Elastic Agent. Can you verify Agent is also root. How did you install Agent?

Output from ps aux | grep elastic-agent should list all instances of Elastic Agent. Also, can you verify only Elastic Agent is listening on ports 6788 and 6789, and determine which of those ports it is listening on. The output from sudo netstat -anp --tcp | grep 678 would help. It might be that some other program is also listening on those ports.

dapster · December 11, 2022, 4:42pm

Agent has been installed originally in 8.5.2 manually thanks to the command provided by Fleet Server.

The 8.5.3 upgrade has been handled automatically by Fleet Server ("Upgrade agent" menu following the upgrade of the whole stack towards 8.5.3).

ps aux | grep elastic-agent leads to (for me everything is run as root):

root        5079  0.1  2.3 1641512 48012 ?       Ssl  Dec09   4:53 /opt/Elastic/Agent/elastic-agent
root        5104  0.0  0.0      0     0 ?        Zs   Dec09   0:00 [elastic-agent] <defunct>
root        5120  0.8  4.2 1544272 86096 ?       Sl   Dec09  24:15 /opt/Elastic/Agent/data/elastic-agent-0e1a73/install/metricbeat-8.5.3-linux-x86_64/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${METRICBEAT_GOGC:100} -E metricbeat.config.modules.enabled=false -E logging.level=info -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/default/metricbeat/metricbeat.sock -E logging.files.path=/opt/Elastic/Agent/data/elastic-agent-0e1a73/logs/default -E logging.files.name=metricbeat -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/opt/Elastic/Agent/data/elastic-agent-0e1a73/run/default/metricbeat--8.5.3
root        5130  1.9  5.4 1701552 111580 ?      Sl   Dec09  56:55 /opt/Elastic/Agent/data/elastic-agent-0e1a73/install/packetbeat-8.5.3-linux-x86_64/packetbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E logging.level=info -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/default/packetbeat/packetbeat.sock -E logging.files.path=/opt/Elastic/Agent/data/elastic-agent-0e1a73/logs/default -E logging.files.name=packetbeat -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/opt/Elastic/Agent/data/elastic-agent-0e1a73/run/default/packetbeat--8.5.3
root        5145  0.8  6.2 1500508 127132 ?      Sl   Dec09  23:09 /opt/Elastic/Agent/data/elastic-agent-0e1a73/install/filebeat-8.5.3-linux-x86_64/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${FILEBEAT_GOGC:100} -E filebeat.config.modules.enabled=false -E logging.level=info -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/default/filebeat/filebeat.sock -E logging.files.path=/opt/Elastic/Agent/data/elastic-agent-0e1a73/logs/default -E logging.files.name=filebeat -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/opt/Elastic/Agent/data/elastic-agent-0e1a73/run/default/filebeat--8.5.3
root        5161  0.1  5.7 1637044 116684 ?      Sl   Dec09   3:04 /opt/Elastic/Agent/data/elastic-agent-0e1a73/install/filebeat-8.5.3-linux-x86_64/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${FILEBEAT_GOGC:100} -E filebeat.config.modules.enabled=false -E logging.level=info -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/default/filebeat/filebeat.sock_monitor -E path.data=/opt/Elastic/Agent/data/elastic-agent-0e1a73/run/default/filebeat--8.5.3--36643631373035623733363936343635
root        5168  0.1  4.0 1469772 82156 ?       Sl   Dec09   3:05 /opt/Elastic/Agent/data/elastic-agent-0e1a73/install/metricbeat-8.5.3-linux-x86_64/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E logging.level=debug -E gc_percent=${METRICBEAT_GOGC:100} -E metricbeat.config.modules.enabled=false -E logging.level=info -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/default/metricbeat/metricbeat.sock_monitor -E path.data=/opt/Elastic/Agent/data/elastic-agent-0e1a73/run/default/metricbeat--8.5.3--36643631373035623733363936343635

sudo netstat -anp --tcp | grep 678 leads to (looks good to me regarding ports associated with elastic-agent):

tcp        0      0 127.0.0.1:6788          0.0.0.0:*               LISTEN      5079/elastic-agent
tcp        0      0 127.0.0.1:6789          0.0.0.0:*               LISTEN      5079/elastic-agent
tcp        0      0 127.0.0.1:31622         127.0.0.1:6789          ESTABLISHED 5145/filebeat
tcp        0      0 127.0.0.1:6789          127.0.0.1:20218         ESTABLISHED 5079/elastic-agent
tcp        0      0 10.78.106.11:26780      10.78.106.11:3128       TIME_WAIT   -
tcp        0      0 127.0.0.1:6789          127.0.0.1:31624         ESTABLISHED 5079/elastic-agent
tcp        0      0 127.0.0.1:6789          127.0.0.1:31622         ESTABLISHED 5079/elastic-agent
tcp        0      0 127.0.0.1:31624         127.0.0.1:6789          ESTABLISHED 5161/filebeat
tcp        0      0 127.0.0.1:28474         127.0.0.1:6789          ESTABLISHED 5130/packetbeat
tcp        0      0 127.0.0.1:6789          127.0.0.1:28472         ESTABLISHED 5079/elastic-agent
tcp        0      0 127.0.0.1:6789          127.0.0.1:28474         ESTABLISHED 5079/elastic-agent
tcp        0      0 127.0.0.1:20218         127.0.0.1:6789          ESTABLISHED 5249/elastic-endpoi
tcp        0      0 127.0.0.1:28472         127.0.0.1:6789          ESTABLISHED 5120/metricbeat
tcp        0      0 127.0.0.1:5436          127.0.0.1:6789          ESTABLISHED 5168/metricbeat
tcp        0      0 127.0.0.1:6789          127.0.0.1:5436          ESTABLISHED 5079/elastic-agent

dapster · December 11, 2022, 4:53pm

Hi Nick,

cat /proc/sys/kernel/ftrace_enabled
1

ferullo · December 14, 2022, 2:53pm

Hmm. I'm not sure why Endpoint is failing to connect to Agent. I'd expected to see something different in the data you shared.

I assume the Endpoint for this host is not showing up in the Security App's Endpoint list, correct? That page is built off documents Endpoint writes to Elasticsearch. If Agent<->Endpoint communication isn't working then Endpoint won't receive information needed to know how to write to Elasticsearch. An alternative way to put this is, does /opt/Elastic/Endpoint/elastic-endpoint.yaml still contain the content at the end of this post? I assume it does, but if it doesn't then the situation you're seeing is different than I understand it to be and that would be good to know.

Are there other Info or Debug logs from Endpoint just before the AgentConnectionInfo.cpp:123 log message?
What does sudo stat /proc/<AGENT_PID>/fd/* output?
Can you attach strace to Endpoint for 20 seconds and then PM me the trace? Endpoint tries to connect to Agent every 10 seconds (and will output AgentComms.cpp:644 Unable to retrieve connection info from Agent(Agent is not running as root each time it fails), I want to see the exact system calls Endpoint is making when that happens. To gather this data use sudo strace -f -p <ENDPOINT_PID>

Here's Endpoint's default configuration it installs with before receiving the "real" configuration from Agent/Fleet.

revision: 0

fleet:
  agent:
    id: 00000000-0000-0000-0000-000000000000
    logging:
      level: info
  host:
    id: 00000000-0000-0000-0000-000000000000

inputs:
  - id: 00000000-0000-0000-0000-000000000000
    name: initial
    revision: 0

dapster · December 26, 2022, 8:09am

Regarding your questions:

Is Endpoint listed in the Security App's Endpoint list: Yes, shown as Healthy
Does the elastic-endpoint.yaml still contain the default configuration: No, various id entries are replaced by actual values, revision is 13
Logs around the AgentConnectionInfo.cpp:123 message:

{"@timestamp":"2022-12-09T07:19:50.366570132Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"
file":{"line":880,"name":"Network.cpp"}}},"message":"Network.cpp:880 Directory [/proc/1866/fd] does not exist?!","process":{"pid":1118,"thread":{"id":1381}}}
{"@timestamp":"2022-12-09T07:19:50.366595808Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"f
ile":{"line":91,"name":"AgentConnectionInfo.cpp"}}},"message":"AgentConnectionInfo.cpp:91 Failed to find connection to validate. Is Agent listening on 127.0.0.1:6788?","process":{"
pid":1118,"thread":{"id":1381}}}
{"@timestamp":"2022-12-09T07:19:50.366605838Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"
file":{"line":123,"name":"AgentConnectionInfo.cpp"}}},"message":"AgentConnectionInfo.cpp:123 Agent process is not root/admin or validation failed, disconnecting","process":{"pid":1
118,"thread":{"id":1381}}}
{"@timestamp":"2022-12-09T07:19:50.366616474Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"warning","origin":
{"file":{"line":180,"name":"AgentConnectionInfo.cpp"}}},"message":"AgentConnectionInfo.cpp:180 Failed to established stage 1 connection to agent","process":{"pid":1118,"thread":{"i
d":1381}}}
{"@timestamp":"2022-12-09T07:19:50.366625674Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"
file":{"line":587,"name":"AgentComms.cpp"}}},"message":"AgentComms.cpp:587 Unable to retrieve connection info from Agent(Agent is not running as root)","process":{"pid":1118,"threa
d":{"id":1381}}}
{"@timestamp":"2022-12-09T07:19:55.435104354Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"f
ile":{"line":563,"name":"LogStorage.cpp"}}},"message":"LogStorage.cpp:563 Unsynced document log store: /opt/Elastic/Endpoint/state/documents/documents-2022-12-08T032453.log","proce
ss":{"pid":1118,"thread":{"id":1334}}}
{"@timestamp":"2022-12-09T07:19:56.511415935Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"info","origin":{"f
ile":{"line":128,"name":"AgentConnectionInfo.cpp"}}},"message":"AgentConnectionInfo.cpp:128 Validated agent (1133) is root/admin","process":{"pid":1118,"thread":{"id":1381}}}

stats /proc/<AGENT_PID>/fd/*

  File: /proc/12000/fd/0 -> /dev/null
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394871    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/1 -> socket:[6673600]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394872    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/10 -> socket:[6672247]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394881    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/12 -> socket:[6693534]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394882    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/13 -> pipe:[6672253]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394883    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/14 -> socket:[6674688]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394884    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/15 -> pipe:[6672254]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394885    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/16 -> pipe:[6672272]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394886    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/17 -> socket:[6674874]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394887    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/18 -> pipe:[6672273]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394888    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/19 -> pipe:[6673881]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394889    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/2 -> socket:[6673600]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394873    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/20 -> pipe:[6673795]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394890    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/21 -> socket:[6674920]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394891    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/22 -> pipe:[6673796]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394892    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/23 -> pipe:[6673882]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394893    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/24 -> pipe:[6673927]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394894    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/25 -> socket:[6673933]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394895    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/26 -> socket:[6674713]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394896    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/27 -> pipe:[6673928]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394897    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/28 -> socket:[43507416]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394898    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/29 -> socket:[6674977]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394899    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/3 -> /opt/Elastic/Agent/data/agent.lock
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394874    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/30 -> socket:[6675009]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394900    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/31 -> socket:[6675014]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394901    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/32 -> socket:[6675055]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394902    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/33 -> socket:[6696211]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394903    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/34 -> socket:[6675282]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394904    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/35 -> socket:[6675285]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394905    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/36 -> socket:[6675286]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394906    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/37 -> socket:[6675289]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394907    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/38 -> socket:[6675292]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394908    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/39 -> socket:[6696214]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394909    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/4 -> anon_inode:[eventpoll]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394875    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/40 -> socket:[6696217]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394910    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/41 -> socket:[6696220]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394911    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/42 -> socket:[6697718]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394912    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/43 -> socket:[6697719]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394913    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/44 -> socket:[6697721]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394914    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/45 -> socket:[6697722]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394915    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/46 -> socket:[6697725]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394916    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/47 -> socket:[7670648]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394917    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/48 -> socket:[7665526]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394918    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/49 -> socket:[7665530]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394919    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/5 -> pipe:[6673631]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394876    Links: 1
Access: (0500/lr-x------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/50 -> socket:[7665533]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394920    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/51 -> socket:[7665536]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394921    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/52 -> socket:[28486881]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394922    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/53 -> socket:[28486884]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394923    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/54 -> socket:[28486887]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394924    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/55 -> socket:[28486891]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394925    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/56 -> socket:[28488568]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394926    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/57 -> socket:[43487176]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43397011    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 07:33:19.510401859 +0000
Modify: 2022-12-26 06:39:54.425670147 +0000
Change: 2022-12-26 06:39:54.425670147 +0000
 Birth: -
  File: /proc/12000/fd/58 -> socket:[43487177]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43485090    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 07:33:19.510401859 +0000
Modify: 2022-12-26 07:33:11.949721444 +0000
Change: 2022-12-26 07:33:11.949721444 +0000
 Birth: -
  File: /proc/12000/fd/59 -> socket:[43487181]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43485091    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 07:33:19.510401859 +0000
Modify: 2022-12-26 07:33:11.949721444 +0000
Change: 2022-12-26 07:33:11.949721444 +0000
 Birth: -
  File: /proc/12000/fd/6 -> pipe:[6673631]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394877    Links: 1
Access: (0300/l-wx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/60 -> socket:[43487182]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43485092    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 07:33:19.510401859 +0000
Modify: 2022-12-26 07:33:11.949721444 +0000
Change: 2022-12-26 07:33:11.949721444 +0000
 Birth: -
  File: /proc/12000/fd/61 -> socket:[43487183]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43485093    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 07:33:19.510401859 +0000
Modify: 2022-12-26 07:33:11.949721444 +0000
Change: 2022-12-26 07:33:11.949721444 +0000
 Birth: -
  File: /proc/12000/fd/7 -> /opt/Elastic/Agent/data/elastic-agent-0e1a73/logs/elastic-agent-20221211-2.ndjson
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394878    Links: 1
Access: (0300/l-wx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/8 -> /opt/Elastic/Agent/elastic-agent-20221211-2.ndjson
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394879    Links: 1
Access: (0300/l-wx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -
  File: /proc/12000/fd/9 -> socket:[6672231]
  Size: 64        	Blocks: 0          IO Block: 1024   symbolic link
Device: 5h/5d	Inode: 43394880    Links: 1
Access: (0700/lrwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-12-26 06:38:59.504717535 +0000
Modify: 2022-12-26 06:38:49.503815681 +0000
Change: 2022-12-26 06:38:49.503815681 +0000
 Birth: -

Strangely, the last log mentioning Unable to retrieve connection info from Agent is quite old and dated of the installation of the 8.5.3:

endpoint-000006.log:{"@timestamp":"2022-12-09T07:19:50.366625674Z","agent":{"id":"bdf61569-dedf-4905-a8a9-207ef1391e8a","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":587,"name":"AgentComms.cpp"}}},"message":"AgentComms.cpp:587 Unable to retrieve connection info from Agent(Agent is not running as root)","process":{"pid":1118,"thread":{"id":1381}}}

I have sent you the trace by PM anyway.

I have double checked the status of elastic-agent diagnostics and there is sill the Get "http://unix/": dial unix /opt/Elastic/Agent/data/tmp/default/endpoint-security/endpoint-security.sock: connect: no such file or directory error.

Thanks

dapster · January 9, 2023, 3:57pm

Following additional testing (and help from @ferullo) the conclusions are:

Endpoint security is working well (a basic anti-malware test works perfectly)
Diagnostics status with an error regarding endpoint-security.sock is confusing but should be ignored

Thanks to all for your help regarding this issue.

system · February 6, 2023, 3:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Problem with Endpoint Security Initiation Endpoint Security	8	1105	November 24, 2022
Elastic Endpoint Security with Elastic Agent Endpoint Security	16	3027	November 10, 2020
Elastic-agent : endpoint-security.sock no such file or directory Beats elastic-agent	4	2338	January 19, 2022
Endpoint 7.13 migration to 7.13.1 Lesson learned with Fleet “On-Prim” -Bad Elastic Security fleet	16	1298	July 21, 2021
Elastic Endpoint cannot connect to agent Endpoint Security	6	299	July 26, 2024

Unable to run Endpoint 8.4 on a Ubuntu 20.04 host hardened with CIS Level 2

Related Topics