No exception lists found

Elastic Security Endpoint Security
1

Hi,

in our default space I get a message saying "No exception lists found" and "We weren't able to find any exception lists." when I get into Exceptions in Security app even tho there are some exceptions (you can even see there should be 2 pages of them)

image
image2302×477 25.9 KB

and here's real example of an existing exception:

image
image2260×497 37.1 KB

I can't think of why this happens only in our default space, when in other spaces/tenants it works properly

image
image2246×389 41 KB

Nothing appears even in the kibana.log, so don't know where to look at or what could be the problem here.

Thanks for any tip or advice.

2

Hi @radovan, could you please let us know which version of Kibana you're running?

Also, could you please send us the output of the following call after you run it on Kibana -> Dev Tools?

// <user-role> should be the role of the user trying to view the Exception lists
GET /_security/role/<user-role>
3

Hi @Pedro_Jaramillo,

the version of elk stack is 7.17.2.

The role doesn't matter here, because i can't see it even with superuser role.

4

Now can't see them on all of the spaces

5

Hi @radovan, could you please run the following query in Kibana -> Dev Tools and let us know if it returns any results? Also, could you please verify that your permissions for .lists index are configured as described in the docs?

GET .kibana*/_search
 {
    "size": 1000,      
    "query": {
     "bool": {
       "must": [
         {
           "exists": {
             "field": "exception-list"
           }
         },
         {
           "query_string": {
             "query": "exception-list.list_type: list"
           }
         }
       ]
     }
   }
 }
6

Yes, it surely returns a few exceptions

"took" : 11376,
  "timed_out" : false,
  "_shards" : {
    "total" : 72,
    "successful" : 72,
    "skipped" : 55,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 853,
      "relation" : "eq"
    },

I can see the exceptions when I open a rule that has one, but can't see them on the Exceptions tab in Security app.

7

Hi @radovan, rule exception lists are visible only in the Kibana space where they were created. That's similar to how rules behave. Rules are also space aware. In your multi-tenant setup, that means that the exception lists created in tenant spaces won't be visible in the "default" space. May I ask why you'd like to see them in the "default" space? Is it because you'd like to manage the lists for all tenants in one place?

8

You still do not get what the problem is. I know how exceptions work. I saw the exceptions (even for default space) in the Exceptions tab and then one day they just didn't load even tho they exist. And I don't know where to look for the cause of this problem.

9

Hi @radovan, I apologize if I misunderstood the problem. The exception lists should not be visible in the "default" space if they were created in a different space. They should be visible in that space if they were created in the same space. Have you created exceptions in the "default" space that you are not able to see in the Exceptions page while in the "default" space?

10

No. I created exceptions in default space that I can see in the default space when I open a specific rule, but not in the Exceptions tab of Security app. Also similar for another spaces where I do have exceptions but can't see them in the Exceptions tab as I shown on the screenshot in original post.

11

Gotcha. Would you mind sending a sample result of an exception list returned by the following query? That could give us additional clues as to what might be going on, as I am not able to reproduce that issue in 7.17.2 currently. Thank you.

GET .kibana*/_search
 {
    "size": 1000,      
    "query": {
     "bool": {
       "must": [
         {
           "exists": {
             "field": "exception-list"
           }
         },
         {
           "query_string": {
             "query": "exception-list.list_type: list"
           }
         }
       ]
     }
   }
 }
12

Hey @radovan !

I'm unable to recreate what you're seeing in 7.17.2, but given that it seems as if the data may be being returned, there may be some filters causing unexpected behavior. Did this page stop working after any Kibana version updates or it was just one day to the next?

If possible, I'd be curious to dissect the API call and what is being returned as you did note that the pagination showed there being data returned. I'm attaching a screenshot of the call that I am looking for specifically (/api/exception_lists/_find?filter=... - please include the entire url that includes the filters). In terms of the data being returned from this call, the fields I'm curious about from the lists returned are type, and namespace_type.

Screen Shot 2022-08-26 at 12.42.35 PM
Screen Shot 2022-08-26 at 12.42.35 PM1920×670 80.8 KB

Thanks for your patience as we try to figure this one out! :slight_smile:

13

Alright, here you go:

{
  "took" : 6351,
  "timed_out" : false,
  "_shards" : {
    "total" : 73,
    "successful" : 73,
    "skipped" : 56,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 854,
      "relation" : "eq"
    },
    "max_score" : 2.1661859,
    "hits" : [
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "XXX:exception-list:072a2390-92ff-11eb-9090-8f2969880775",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2021-04-01T15:29:19.945Z",
            "created_by" : "XXX",
            "description" : "Any interaction with dionaea honeypot.",
            "immutable" : false,
            "list_id" : "5cd489a6-4533-44ff-b38a-bc5d93052915",
            "list_type" : "list",
            "name" : "Dionaea interaction",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "2e17679f-a1f7-4318-be28-9022265fd597",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "namespace" : "XXX",
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2021-04-01T15:29:19.952Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "XXX:exception-list:b37ac880-2b4c-11ec-b68a-ab868ee1c135",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2021-10-12T11:08:17.031Z",
            "created_by" : "XXX",
            "description" : "Adversaries may attempt to get information about running processes on a system.",
            "immutable" : false,
            "list_id" : "76288ec6-4e6b-4c52-9088-5761aa52293f",
            "list_type" : "list",
            "name" : "Process Discovery via Tasklist",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "3984f513-b499-45e8-91f3-046beb6c4715",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "namespace" : "XXX",
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2021-10-12T11:08:17.046Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "XXX:exception-list:0c8f93c0-29a9-11eb-a164-fd3748a12a8e",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "_tags" : [ ],
            "created_at" : "2020-11-18T14:19:20.183Z",
            "created_by" : "XXX",
            "description" : "Malware DNS Trap is used to identify compromised clients attempting to access known malicious domains. When this feature is enabled, gateway does not block DNS requests that were identified as malicious. The response is tampered and a false (bogus) IP address is returned to the client. Using the Malware DNS Trap you can then detect compromised clients by checking logs with connection attempts to the false IP address. Consecutive connections addressed to the bogus IP are blocked. The default value for DNS trap IP is 62.0.58.94. ",
            "immutable" : false,
            "list_id" : "f4b1b75a-87fb-4d08-bfb7-02488ebdbb6e",
            "list_type" : "list",
            "name" : "Checkpoint DNS trap",
            "tags" : [ ],
            "tie_breaker_id" : "e7b5a7a1-1239-4fa8-a792-a73fe639e19f",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "namespace" : "XXX",
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2020-11-18T14:19:20.188Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "XXX:exception-list:67d42a60-01cb-11ed-87b0-a5d4c1d9b2c5",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2022-07-12T10:14:25.030Z",
            "created_by" : "XXX",
            "description" : "Identifies an unexpected file being modified by dns.exe, the process responsible for Windows DNS Server services, which may indicate activity related to remote code execution or other forms of exploitation.",
            "immutable" : false,
            "list_id" : "a3cec274-98ad-4bac-8eaa-137f15726b46",
            "list_type" : "list",
            "name" : "Unusual File Modification by dns.exe",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "991caf08-080a-4eb8-82f2-961032fe1043",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "namespace" : "XXX",
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2022-07-12T10:14:25.037Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "XXX:exception-list:4fb5fc40-d5e0-11ec-82db-49f2e96bbcbc",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2022-05-17T12:53:12.836Z",
            "created_by" : "XXX",
            "description" : """Detects the access to processes by other suspicious processes which have reflectively loaded libraries in their memory space. An example is SilentTrinity C2 behaviour. Generally speaking, when Sysmon EventID 10 cannot reference a stack call to a dll loaded from disk (the standard way), it will display "UNKNOWN" as the module name. Usually this means the stack call points to a module that was reflectively loaded in memory. Adding to this, it is not common to see such few calls in the stack (ntdll.dll --> kernelbase.dll --> unknown) which essentially means that most of the functions required by the process to execute certain routines are already present in memory, not requiring any calls to external libraries. The latter should also be considered suspicious.""",
            "immutable" : false,
            "list_id" : "457831c8-9cb6-4c7d-9093-14990062a4cf",
            "list_type" : "list",
            "name" : "Suspicious In-Memory Module Execution",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "f20ca6c1-cd25-4f5e-bfb4-26a1b56c39f7",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "namespace" : "XXX",
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2022-05-17T12:53:12.844Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "XXX:exception-list:57134330-d5e5-11ec-82db-49f2e96bbcbc",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2022-05-17T13:29:12.674Z",
            "created_by" : "XXX",
            "description" : """This rule detects network events that may indicate the use of SSH traffic
from the Internet. SSH is commonly used by system administrators to remotely
control a system using the command line shell. If it is exposed to the Internet,
it should be done with strong security controls as it is frequently targeted and
exploited by threat actors as an initial access or back-door vector.
""",
            "immutable" : false,
            "list_id" : "f946a9cd-129f-4e43-b7ee-a145b1e1de03",
            "list_type" : "list",
            "name" : "SSH (Secure Shell) to the Internet",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "72599801-9312-4ec5-94d5-61568bff0bcb",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "namespace" : "XXX",
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2022-05-17T13:29:12.683Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "exception-list:887927b0-882d-11ec-beea-cf06de1ba3b9",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2022-02-07T15:49:28.619Z",
            "created_by" : "XXX",
            "description" : """This rule detects events that may indicate use of a PPTP VPN connection. Some
threat actors use these types of connections to tunnel their traffic while
avoiding detection.
""",
            "immutable" : false,
            "list_id" : "461af834-d78b-4218-90a2-58f893166005",
            "list_type" : "list",
            "name" : "PPTP (Point to Point Tunneling Protocol) Activity",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "1da66c06-70c2-4bc4-a8bd-b3d1b3e197d2",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2022-02-07T15:49:28.627Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "XXX:exception-list:9534bce0-e1be-11ec-87b0-a5d4c1d9b2c5",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2022-06-01T15:22:00.494Z",
            "created_by" : "XXX",
            "description" : """This rule detects events that may describe SMTP traffic from internal
hosts to a host across the Internet. In an enterprise network, there is typically
a dedicated internal host that performs this function. It is also
frequently abused by threat actors for command and control, or data exfiltration.
""",
            "immutable" : false,
            "list_id" : "0dc2ccb8-9680-4883-a546-2a1bca188deb",
            "list_type" : "list",
            "name" : "SMTP to the Internet",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "14c899c5-0448-4a72-8eef-321783c52233",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "namespace" : "XXX",
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2022-06-01T15:22:00.502Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "XXX:exception-list:c162e610-e1bf-11ec-87b0-a5d4c1d9b2c5",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2022-06-01T15:30:24.113Z",
            "created_by" : "XXX",
            "description" : """This rule detects events that may describe database traffic
(MS SQL, Oracle, MySQL, and Postgresql) across the Internet. Databases
should almost never be directly exposed to the Internet, as they are
frequently targeted by threat actors to gain initial access to network resources.
""",
            "immutable" : false,
            "list_id" : "f9b00789-6f61-4027-b7e7-630ffbd445b1",
            "list_type" : "list",
            "name" : "SQL Traffic to the Internet",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "6b3ea150-bf80-4d6b-91a9-4796e8f8bc86",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "namespace" : "XXX",
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2022-06-01T15:30:24.124Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "XXX:exception-list:d8122140-e1c0-11ec-87b0-a5d4c1d9b2c5",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2022-06-01T15:38:11.668Z",
            "created_by" : "XXX",
            "description" : """This rule detects events that use common ports for Internet Relay Chat (IRC) to the Internet.
IRC is a common protocol that can be used for chat and file transfers. This
protocol is also a good candidate for remote control of malware and data
transfers to and from a network.
""",
            "immutable" : false,
            "list_id" : "eb029e9f-6a9b-404a-9aa9-d35ee38f1fd7",
            "list_type" : "list",
            "name" : "IRC (Internet Relay Chat) Protocol Activity to the Internet",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "31d1e959-cc17-49ee-93ef-9584c4e2eaf9",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "namespace" : "XXX",
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2022-06-01T15:38:11.674Z"
        }
      },
      {
        "_index" : ".kibana_7.17.2_001",
        "_type" : "_doc",
        "_id" : "exception-list:0b8342a0-e277-11ec-87b0-a5d4c1d9b2c5",
        "_score" : 2.1661859,
        "_source" : {
          "exception-list" : {
            "created_at" : "2022-06-02T13:22:26.378Z",
            "created_by" : "XXX",
            "description" : "Adversaries may attempt to disable the iptables or firewall service in an attempt to affect how a host is allowed to receive or send network traffic.",
            "immutable" : false,
            "list_id" : "a1d95c57-503b-42f4-a5fe-fb64bc001406",
            "list_type" : "list",
            "name" : "Attempt to Disable IPTables or Firewall",
            "os_types" : [ ],
            "tags" : [ ],
            "tie_breaker_id" : "55c8b276-4ab2-43ff-96e5-5354160669be",
            "type" : "detection",
            "updated_by" : "XXX",
            "version" : 1
          },
          "type" : "exception-list",
          "references" : [ ],
          "migrationVersion" : {
            "exception-list" : "7.12.0"
          },
          "coreMigrationVersion" : "7.17.2",
          "updated_at" : "2022-06-02T13:22:26.384Z"
        }
      },

and so on...

14

Hi @yctercero,

so the are 2 requests like in your screenshot, one that is failed (400) with url path:
/api/detection_engine/rules/_find?page=1&per_page=10000&sort_field=enabled&sort_order=desc
which returns:
"all shards failed: search_phase_execution_exception: [too_many_clauses] Reason: too_many_clauses: maxClauseCount is set to 1024"

and second one with url path /api/exception_lists/_find?filter=(not%20exception-list.attributes.list_id%3A%20endpoint_trusted_apps*%20AND%20not%20exception-list-agnostic.attributes.list_id%3A%20endpoint_trusted_apps*)%20AND%20(not%20exception-list.attributes.list_id%3A%20endpoint_event_filters*%20AND%20not%20exception-list-agnostic.attributes.list_id%3A%20endpoint_event_filters*)%20AND%20(not%20exception-list.attributes.list_id%3A%20endpoint_host_isolation_exceptions*%20AND%20not%20exception-list-agnostic.attributes.list_id%3A%20endpoint_host_isolation_exceptions*)&namespace_type=single%2Cagnostic&page=1&per_page=20&sort_field=exception-list.created_at&sort_order=desc)

that actually returns data and it's namespace_type is single and type is detection for all of the returned data.

And sorry for the late reply, I was on a vacation :slight_smile:

15

Hey Radovan!

Hope you had a good vacation :slight_smile:

So after chatting with a colleague, there's a few things that could be causing that 400 response and error message you're seeing. That response is coming from Elasticsearch and can occur if any single one of your shards are in bad health. I don't suspect it's this as I would also then think that other endpoints would be failing. However, you can double check this by using the _health endpoint documented here. You can make the following query from the Kibana dev tools: GET _cluster/health.

The other cause of that error is if the formatted query is excessively large. I would only expect this to be the case if you have many thousands of rules. So curious, how many rules are you dealing with in your space and overall? This call should be scoped to the space but curious to know if across spaces the number of rules adds up to be very large.

We weren't able to recreate this error in 8.4 even with 10000 rules, however, we did recreate this error in 7.17 when testing with 5000 rules. While I haven't been able to confirm, I assume that post 7.17, updates were made to how we format the query to fetch rules to avoid this very error. I'm still curious to hear how many rules you are working with and see if I can recreate what you're seeing.

Best,
Yara

1 Like
16

Hi Yara,

the cluster health is fine, I monitor that frequently.

The number of rules across spaces do not exceed the total of 10k rules. But it really looks like that in spaces, where I got a little more than 1k rules I cannot see exceptions and in spaces that has less than 1k rules the exceptions work fine.

Seems like i finally have to make that upgrade for 8.X.

Thank you @yctercero :slight_smile:

17

Hi @radovan, I saw this thread yesterday and dug into our changelog today. We fixed the exact problem you're facing in 8.0.0, here's the corresponding GitHub issue:

Seems like i finally have to make that upgrade for 8.X.

That would be great! :slight_smile: Please note that there's still an outstanding issue with having a large number of rules in a space:

Hope that helps. Thank you for all the details in this thread.

3 Likes
18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.