Calling Alerts from Watchers to detection Signals

Suro · September 25, 2020, 10:12pm

Hi,
We are running multiple watchers and as the SIEM detection does not provide the level of customizations, I have been unable to create use cases which takes aggregation for example under consideration. So I want to create a rule in SIEM that it has to check the alerts index every 5 mins and generate a signal with the alerts observed.

Something that I am trying - _index: alerts*. The Signal Rule shows successfully executed but no events gets generated even when the alerts index is getting the alerts on time.

Frank_Hassanabad · September 25, 2020, 11:35pm

We do currently have threshold rules that might help you out?

But if not your current issue could be that the alert index is not compatible with the ECS (Elastic Common Schema) and then cannot copy over the signal.

Do you have the mapping of the alert index you are trying to create a signal from? I can take a look for you. We have improved the error messaging in our next upcoming release to report more errors transparently to the UI from this release fwiw:

borna_talebi · September 26, 2020, 4:46am

Hi,
Did you add the index pattern into siem default indices? (via stack management->kibana-> advanced setting)
If you did, it should be available in the index patterns section when you want to create a new rule.
also, try a different query like somefield : * and see if it works.

Suro · September 26, 2020, 10:37am

I have added alerts* under default indices and still not working -

Here is the use case -

When I run the same query in Discover Kibana, I get the events, but the Detection rule is not letting me do it.

I would need help with the other suggestions from Frank -
a) How do I check the mapping of the index. This is what I saw, let me know if this helps -

b) The option for threshold that is displayed does not allow me to make use cases based on Unique values. For Example, here are some use cases -

One User Account used to login to multiple machines in a short span of time.
Single IP making multiple connections to other machines within the network via SMB/ telnet etc.
BruteForce Attempts

Please suggest if Watchers are not required for these Use Cases.

borna_talebi · September 26, 2020, 11:12am

instead of _index : "alerts*" try something like agent.name : * and see if it works.

you can also use mapping API for that.

Suro · September 26, 2020, 5:10pm

Tried that ....did not work -

Suro · September 26, 2020, 5:15pm

my bad the field name Agent doesnot exists in Alerts index. So I am trying one Use case specifically and trying to see if that is generated!!!!

Suro · September 26, 2020, 5:44pm

That did not work-

here is the discover query-

With the same parameters, here is the Detection rule -

Still no detections.

Frank_Hassanabad · September 28, 2020, 1:01pm

What's the mapping? Run something like this below in your dev tools and see what it is. The signals index mapping is that of ECS, so if there is a conflict or issue where the source index which is your alerts in this case is too far away from ECS then it more than likely is not going to work out for you.

In dev tools you can do:

GET /alerts*/_mapping

And then give us the output and we can see.

Suro · September 28, 2020, 1:15pm

Here is the output -

{
  "alerts" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "agent" : {
          "properties" : {
            "hostname" : {
              "type" : "keyword"
            },
            "name" : {
              "type" : "keyword"
            },
            "version" : {
              "type" : "keyword"
            }
          }
        },
        "alert_id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "alert_instance_id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "alert_name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "beat" : {
          "properties" : {
            "hostname" : {
              "type" : "keyword"
            },
            "name" : {
              "type" : "keyword"
            },
            "version" : {
              "type" : "keyword"
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "keyword"
            }
          }
        },
        "faied_count_alert" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "host" : {
          "properties" : {
            "architecture" : {
              "type" : "keyword"
            },
            "containerized" : {
              "type" : "boolean"
            },
            "geo" : {
              "properties" : {
                "city_name" : {
                  "type" : "keyword"
                },
                "continent_name" : {
                  "type" : "keyword"
                },
                "country_iso_code" : {
                  "type" : "keyword"
                },
                "country_name" : {
                  "type" : "keyword"
                },
                "location" : {
                  "type" : "geo_point"
                },
                "name" : {
                  "type" : "keyword"
                },
                "region_iso_code" : {
                  "type" : "keyword"
                },
                "region_name" : {
                  "type" : "keyword"
                }
              }
            },
            "hostname" : {
              "type" : "keyword"
            },
            "id" : {
              "type" : "keyword"
            },
            "ip" : {
              "type" : "ip"
            },
            "mac" : {
              "type" : "keyword"
            },
            "name" : {
              "type" : "keyword"
            },
            "os" : {
              "properties" : {
                "build" : {
                  "type" : "keyword"
                },
                "codename" : {
                  "type" : "keyword"
                },
                "family" : {
                  "type" : "keyword"
                },
                "full" : {
                  "type" : "keyword"
                },
                "kernel" : {
                  "type" : "keyword"
                },
                "name" : {
                  "type" : "keyword"
                },
                "platform" : {
                  "type" : "keyword"
                },
                "version" : {
                  "type" : "keyword"
                }
              }
            },
            "type" : {
              "type" : "keyword"
            },
            "user" : {
              "properties" : {
                "email" : {
                  "type" : "keyword"
                },
                "full_name" : {
                  "type" : "keyword"
                },
                "group" : {
                  "properties" : {
                    "id" : {
                      "type" : "keyword"
                    },
                    "name" : {
                      "type" : "keyword"
                    }
                  }
                },
                "hash" : {
                  "type" : "keyword"
                },
                "id" : {
                  "type" : "keyword"
                },
                "name" : {
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "host_count" : {
          "type" : "long"
        },
        "host_name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "keyword"
                }
              }
            },
            "flags" : {
              "type" : "keyword"
            },
            "level" : {
              "type" : "keyword"
            },
            "offset" : {
              "type" : "long"
            },
            "source" : {
              "properties" : {
                "address" : {
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "message" : {
          "type" : "text"
        },
        "severity" : {
          "type" : "long"
        },
        "source_host" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source_host_alert" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source_ip" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source_user" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source_user_alert" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "total_count" : {
          "type" : "long"
        },
        "triggered" : {
          "type" : "date"
        },
        "user_name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "watch_id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

Frank_Hassanabad · September 29, 2020, 5:27pm

Don't see any mapping conflicts with it at the moment. If you have access to your Kibana logs do you see any errors when the signals are being run?

Suro · September 30, 2020, 1:23pm

No Errors -

message:

alert executed: siem.signals:0eeb4d4c-2ed9-4098-91ca-6ae1a72d3720: 'Watcher Alerts'

 

event.provider:

alerting

 

event.action:

execute

 

event.start:

Sep 30, 2020 @ 12:56:29.207

 

event.end:

Sep 30, 2020 @ 12:56:31.056

 

event.duration:

1,849,000,000

 

event.outcome:

success

 

kibana.saved_objects:

{ "rel": "primary", "type": "alert", "id": "0eeb4d4c-2ed9-4098-91ca-6ae1a72d3720" }

 

kibana.server_uuid:

db28e422-3b70-4b7d-afbd-1c94869e3914

 

@timestamp:

Sep 30, 2020 @ 12:56:31.056

 

ecs.version:

1.5.0

 

_id:

bZgV33QBf6VWBkBR8PlR

 

_type:

_doc

 

_index:

.kibana-event-log-7.9.0-000002

 

_score:

 -

Frank_Hassanabad · September 30, 2020, 1:59pm

You can put Kibana into a type of debug mode like so in kibana.yml to get more log output to help out:

logging.verbose: true
logging.events:
  {
    log: ['securitySolution', 'alerting', info', 'warning', 'error', 'fatal'],
    request: ['info', 'warning', 'error', 'fatal'],
    error: '*',
    ops: __no-ops__,
  }

Suro · September 30, 2020, 10:11pm

Interestingly I am seeing the same issue with another index related to defender events. Though I will try to get the yml file updated, did u see anyone else complaining about this.

Suro · October 1, 2020, 8:36am

Still getting the same. The setting suggested was performed but the logs still says success but no alert -

{
  "_index": ".kibana-event-log-7.9.0-000002",
  "_type": "_doc",
  "_id": "HXhF43QBf6VWBkBR6f7K",
  "_score": 1,
  "_source": {
    "event": {
      "provider": "alerting",
      "action": "execute",
      "start": "2020-10-01T08:27:22.474Z",
      "end": "2020-10-01T08:27:23.978Z",
      "duration": 1504000000,
      "outcome": "success"
    },
    "kibana": {
      "saved_objects": [
        {
          "rel": "primary",
          "type": "alert",
          "id": "0eeb4d4c-2ed9-4098-91ca-6ae1a72d3720"
        }
      ],
      "server_uuid": "db28e422-3b70-4b7d-afbd-1c94869e3914"
    },
    "message": "alert executed: siem.signals:0eeb4d4c-2ed9-4098-91ca-6ae1a72d3720: 'Watcher Alerts'",
    "@timestamp": "2020-10-01T08:27:23.978Z",
    "ecs": {
      "version": "1.5.0"
    }
  },
  "fields": {
    "event.start": [
      "2020-10-01T08:27:22.474Z"
    ],
    "event.end": [
      "2020-10-01T08:27:23.978Z"
    ],
    "@timestamp": [
      "2020-10-01T08:27:23.978Z"
    ]
  }
}

Similar problem with another index - windowsdefender

{
  "_index": ".kibana-event-log-7.9.0-000002",
  "_type": "_doc",
  "_id": "L4ZL43QBf6VWBkBRV1S0",
  "_score": 1,
  "_source": {
    "event": {
      "provider": "alerting",
      "action": "execute",
      "start": "2020-10-01T08:33:17.647Z",
      "end": "2020-10-01T08:33:19.796Z",
      "duration": 2149000000,
      "outcome": "success"
    },
    "kibana": {
      "saved_objects": [
        {
          "rel": "primary",
          "type": "alert",
          "id": "caa921df-65f1-492c-9923-b24ef5271dc3"
        }
      ],
      "server_uuid": "db28e422-3b70-4b7d-afbd-1c94869e3914"
    },
    "message": "alert executed: siem.signals:caa921df-65f1-492c-9923-b24ef5271dc3: 'Virus Infected Machines (Quarantine Failure) - SNOW'",
    "@timestamp": "2020-10-01T08:33:19.796Z",
    "ecs": {
      "version": "1.5.0"
    }
  },
  "fields": {
    "event.start": [
      "2020-10-01T08:33:17.647Z"
    ],
    "event.end": [
      "2020-10-01T08:33:19.796Z"
    ],
    "@timestamp": [
      "2020-10-01T08:33:19.796Z"
    ]
  }
}

system · October 29, 2020, 8:36am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Detection rules SIEM detection-rules	4	681	January 11, 2021
Detection Engine does not create Signals anymore Elastic Security detection-rules	1	576	December 1, 2021
Failing to get Detection Alerts SIEM detection-rules	2	530	February 24, 2022
Alerting by amount of "hits" SIEM	2	404	June 18, 2020
Signal SIEM Detections using log files SIEM	5	430	May 23, 2020

Calling Alerts from Watchers to detection Signals

Related topics