Endpoint Security DEGRADED, Malware failed to enable due to potential system deadlock

[c1jt]

Hi

I have a node with endpoint-security policy applied, but is set to "DEGRADED" and "Unhealthy" after policy is applied/updated. Apparantly due to "potential system deadlock"

Security Endpoints says:

> Malware
>> Configure Malware: Failed to enable malware detection/prevention
>> Load Malware Model: Disabled due to potential system deadlock

Here's a picture for better lookyness

This did work all yesterday with no issues, and only stopped working after I enabled the Sophos integration and later restarted it. I've since tried removing the Sophos integration & fully restarting, but this didn't solve the issue. Haven't made any changes to the endpoints integration.

Anyone have ideas to try?

ty

[lesio (Leszek Kubik)]

Hello, could you tell us what is the OS you've enrolled and what's the Endpoint version?

[c1jt]

Hi,
Yes of course, I should have mentioned;

Everything elastic is currently on version 7.14.0, Endpoint Security is version v0.20.2

This particular endpoint is Debian 10:

Linux #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux

Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

Thanks,

[c1jt]

Riiiiiiiiiiiight so this seems to have fixed itself over the weekend... I just left it to sit (semi-production lol) in this state and Monday morning all is green

Probably some environmental state mismatch, will keep an eye on it.

[lesio (Leszek Kubik)]

I'm glad to hear your system works well now

We would like to examine the endpoint logs to check if there's any explanation what has happened. Could you provide us the files located at /opt/Elastic/Endpoint/state/log ?

Thanks,

[c1jt]

Thanks!

Here's a part which I think is relevent, it does a policy update and goes through each intergration, then it gets to this bit:

...
{"@timestamp":"2021-08-13T08:05:13.60422839Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action load_malware_model: success - Successfully loaded malware model","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.604257245Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action load_diagnostic_malware_model: success - Successfully loaded malware model","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.60770772Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":131,"name":"WriteSuppressionCache.cpp"}}},"message":"WriteSuppressionCache.cpp:131 Clearing the write suppression cache","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.627313558Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"warning","origin":{"file":{"line":141,"name":"YaraLib.cpp"}}},"message":"YaraLib.cpp:141 Rules identifier [filescore-diagnostic] already found, removing it","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.631896216Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action load_diagnostic_malware_model: success - Successfully loaded malware model","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.649559728Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"warning","origin":{"file":{"line":141,"name":"YaraLib.cpp"}}},"message":"YaraLib.cpp:141 Rules identifier [filescore-production] already found, removing it","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654126886Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action load_malware_model: success - Successfully loaded malware model","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654225067Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action load_diagnostic_malware_model: failure - Disabled due to potential system deadlock","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654243004Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action load_malware_model: failure - Disabled due to potential system deadlock","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654260679Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action configure_malware: failure - Failed to enable malware detection/prevention","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654278057Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action configure_diagnostic_malware: failure - Failed to enable malware detection/prevention","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654292071Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":1536,"name":"Config.cpp"}}},"message":"Config.cpp:1536 Configuring qa","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654304982Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":1536,"name":"Config.cpp"}}},"message":"Config.cpp:1536 Configuring endUserNotification","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654326855Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action configure_user_notification: success - Successfully configured user notification","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654336841Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":1536,"name":"Config.cpp"}}},"message":"Config.cpp:1536 Configuring rulesEngine","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654413435Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action configure_diagnostic_behavior_protection: success - Enabled 4/4 diagnostic rules","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654422971Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":1536,"name":"Config.cpp"}}},"message":"Config.cpp:1536 Configuring hostIsolation","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654442351Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action configure_host_isolation: unsupported - Host isolation is not supported on Linux","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654452023Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":1597,"name":"Config.cpp"}}},"message":"Config.cpp:1597 Checking for agent connectivity","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654460917Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":1615,"name":"Config.cpp"}}},"message":"Config.cpp:1615 Checking for agent connectivity","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654478216Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action agent_connectivity: success - Successfully connected to Agent","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654498665Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":365,"name":"Response.cpp"}}},"message":"Response.cpp:365 Policy action workflow: success - Successfully executed all workflows","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654626323Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":574,"name":"Response.cpp"}}},"message":"Response.cpp:574 Setting malware to failure because of configure_malware status","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654643595Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":574,"name":"Response.cpp"}}},"message":"Response.cpp:574 Setting malware to failure because of configure_diagnostic_malware status","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654668281Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":574,"name":"Response.cpp"}}},"message":"Response.cpp:574 Setting host_isolation to unsupported because of read_host_isolation_config status","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654696099Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":1682,"name":"Config.cpp"}}},"message":"Config.cpp:1682 Failed to apply or enrich policy result","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.654716577Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"error","origin":{"file":{"line":1785,"name":"Config.cpp"}}},"message":"Config.cpp:1785 Policy failed to apply and rollback is disabled","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.655850342Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":138,"name":"Metadata.cpp"}}},"message":"Metadata.cpp:138 Sending off-schedule metadata message","process":{"pid":22015,"thread":{"id":22776}}}
{"@timestamp":"2021-08-13T08:05:13.655876987Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":142,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:142 Reconfigure detected, refreshing Elasticsearch client","process":{"pid":22015,"thread":{"id":22022}}}
{"@timestamp":"2021-08-13T08:05:13.655899401Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"info","origin":{"file":{"line":52,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:52 Setting new Elasticsearch client","process":{"pid":22015,"thread":{"id":22022}}}
{"@timestamp":"2021-08-13T08:05:13.655932513Z","agent":{"id":"faa6accf-f760-4d35-ae0b-525bda814523","type":"endpoint"},"ecs":{"version":"1.6.0"},"log":{"level":"error","origin":{"file":{"line":381,"name":"AgentComms.cpp"}}},"message":"AgentComms.cpp:381 Failed to apply new policy from Agent.","process":{"pid":22015,"thread":{"id":22776}}}

Soon after that it starts sending documents to Elasticsearch like normal.

The full logs files are ~25MB each but only 2 are in the right date range, if you would like the entire log file let me know and I'll PM them to you.

Cheers,

[lesio (Leszek Kubik)]

Hi, I was also looking for errors related to fanotify and log about mount points. For the latter the output from cat /proc/mount would provide the current state. Could you post snippets from the logs around fanotify keyword, or PM me the full logs?

[c1jt]

Hi,
I'll PM you the full logs on mega, and output from mounts file.