Calling Alerts from Watchers to detection Signals

Hi,
We are running multiple watchers and as the SIEM detection does not provide the level of customizations, I have been unable to create use cases which takes aggregation for example under consideration. So I want to create a rule in SIEM that it has to check the alerts index every 5 mins and generate a signal with the alerts observed.

Something that I am trying - _index: alerts*. The Signal Rule shows successfully executed but no events gets generated even when the alerts index is getting the alerts on time.

We do currently have threshold rules that might help you out?

But if not your current issue could be that the alert index is not compatible with the ECS (Elastic Common Schema) and then cannot copy over the signal.

Do you have the mapping of the alert index you are trying to create a signal from? I can take a look for you. We have improved the error messaging in our next upcoming release to report more errors transparently to the UI from this release fwiw:


Hi,
Did you add the index pattern into siem default indices? (via stack management->kibana-> advanced setting)
If you did, it should be available in the index patterns section when you want to create a new rule.
also, try a different query like somefield : * and see if it works.

I have added alerts* under default indices and still not working -

Here is the use case -

When I run the same query in Discover Kibana, I get the events, but the Detection rule is not letting me do it.

I would need help with the other suggestions from Frank -
a) How do I check the mapping of the index. This is what I saw, let me know if this helps -

b) The option for threshold that is displayed does not allow me to make use cases based on Unique values. For Example, here are some use cases -

  1. One User Account used to login to multiple machines in a short span of time.
  2. Single IP making multiple connections to other machines within the network via SMB/ telnet etc.
  3. BruteForce Attempts

Please suggest if Watchers are not required for these Use Cases.

instead of _index : "alerts*" try something like agent.name : * and see if it works.

you can also use mapping API for that.

Tried that ....did not work -

my bad the field name Agent doesnot exists in Alerts index. So I am trying one Use case specifically and trying to see if that is generated!!!!

That did not work-

here is the discover query-

With the same parameters, here is the Detection rule -

Still no detections.

What's the mapping? Run something like this below in your dev tools and see what it is. The signals index mapping is that of ECS, so if there is a conflict or issue where the source index which is your alerts in this case is too far away from ECS then it more than likely is not going to work out for you.

In dev tools you can do:

GET /alerts*/_mapping

And then give us the output and we can see.

Here is the output -

{
  "alerts" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "agent" : {
          "properties" : {
            "hostname" : {
              "type" : "keyword"
            },
            "name" : {
              "type" : "keyword"
            },
            "version" : {
              "type" : "keyword"
            }
          }
        },
        "alert_id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "alert_instance_id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "alert_name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "beat" : {
          "properties" : {
            "hostname" : {
              "type" : "keyword"
            },
            "name" : {
              "type" : "keyword"
            },
            "version" : {
              "type" : "keyword"
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "keyword"
            }
          }
        },
        "faied_count_alert" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "host" : {
          "properties" : {
            "architecture" : {
              "type" : "keyword"
            },
            "containerized" : {
              "type" : "boolean"
            },
            "geo" : {
              "properties" : {
                "city_name" : {
                  "type" : "keyword"
                },
                "continent_name" : {
                  "type" : "keyword"
                },
                "country_iso_code" : {
                  "type" : "keyword"
                },
                "country_name" : {
                  "type" : "keyword"
                },
                "location" : {
                  "type" : "geo_point"
                },
                "name" : {
                  "type" : "keyword"
                },
                "region_iso_code" : {
                  "type" : "keyword"
                },
                "region_name" : {
                  "type" : "keyword"
                }
              }
            },
            "hostname" : {
              "type" : "keyword"
            },
            "id" : {
              "type" : "keyword"
            },
            "ip" : {
              "type" : "ip"
            },
            "mac" : {
              "type" : "keyword"
            },
            "name" : {
              "type" : "keyword"
            },
            "os" : {
              "properties" : {
                "build" : {
                  "type" : "keyword"
                },
                "codename" : {
                  "type" : "keyword"
                },
                "family" : {
                  "type" : "keyword"
                },
                "full" : {
                  "type" : "keyword"
                },
                "kernel" : {
                  "type" : "keyword"
                },
                "name" : {
                  "type" : "keyword"
                },
                "platform" : {
                  "type" : "keyword"
                },
                "version" : {
                  "type" : "keyword"
                }
              }
            },
            "type" : {
              "type" : "keyword"
            },
            "user" : {
              "properties" : {
                "email" : {
                  "type" : "keyword"
                },
                "full_name" : {
                  "type" : "keyword"
                },
                "group" : {
                  "properties" : {
                    "id" : {
                      "type" : "keyword"
                    },
                    "name" : {
                      "type" : "keyword"
                    }
                  }
                },
                "hash" : {
                  "type" : "keyword"
                },
                "id" : {
                  "type" : "keyword"
                },
                "name" : {
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "host_count" : {
          "type" : "long"
        },
        "host_name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "keyword"
                }
              }
            },
            "flags" : {
              "type" : "keyword"
            },
            "level" : {
              "type" : "keyword"
            },
            "offset" : {
              "type" : "long"
            },
            "source" : {
              "properties" : {
                "address" : {
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "message" : {
          "type" : "text"
        },
        "severity" : {
          "type" : "long"
        },
        "source_host" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source_host_alert" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source_ip" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source_user" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source_user_alert" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "total_count" : {
          "type" : "long"
        },
        "triggered" : {
          "type" : "date"
        },
        "user_name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "watch_id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

Don't see any mapping conflicts with it at the moment. If you have access to your Kibana logs do you see any errors when the signals are being run?

No Errors -

message:

alert executed: siem.signals:0eeb4d4c-2ed9-4098-91ca-6ae1a72d3720: 'Watcher Alerts'

 

event.provider:

alerting

 

event.action:

execute

 

event.start:

Sep 30, 2020 @ 12:56:29.207

 

event.end:

Sep 30, 2020 @ 12:56:31.056

 

event.duration:

1,849,000,000

 

event.outcome:

success

 

kibana.saved_objects:

{ "rel": "primary", "type": "alert", "id": "0eeb4d4c-2ed9-4098-91ca-6ae1a72d3720" }

 

kibana.server_uuid:

db28e422-3b70-4b7d-afbd-1c94869e3914

 

@timestamp:

Sep 30, 2020 @ 12:56:31.056

 

ecs.version:

1.5.0

 

_id:

bZgV33QBf6VWBkBR8PlR

 

_type:

_doc

 

_index:

.kibana-event-log-7.9.0-000002

 

_score:

 -

You can put Kibana into a type of debug mode like so in kibana.yml to get more log output to help out:

logging.verbose: true
logging.events:
  {
    log: ['securitySolution', 'alerting', info', 'warning', 'error', 'fatal'],
    request: ['info', 'warning', 'error', 'fatal'],
    error: '*',
    ops: __no-ops__,
  }

Interestingly I am seeing the same issue with another index related to defender events. Though I will try to get the yml file updated, did u see anyone else complaining about this.

Still getting the same. The setting suggested was performed but the logs still says success but no alert -

{
  "_index": ".kibana-event-log-7.9.0-000002",
  "_type": "_doc",
  "_id": "HXhF43QBf6VWBkBR6f7K",
  "_score": 1,
  "_source": {
    "event": {
      "provider": "alerting",
      "action": "execute",
      "start": "2020-10-01T08:27:22.474Z",
      "end": "2020-10-01T08:27:23.978Z",
      "duration": 1504000000,
      "outcome": "success"
    },
    "kibana": {
      "saved_objects": [
        {
          "rel": "primary",
          "type": "alert",
          "id": "0eeb4d4c-2ed9-4098-91ca-6ae1a72d3720"
        }
      ],
      "server_uuid": "db28e422-3b70-4b7d-afbd-1c94869e3914"
    },
    "message": "alert executed: siem.signals:0eeb4d4c-2ed9-4098-91ca-6ae1a72d3720: 'Watcher Alerts'",
    "@timestamp": "2020-10-01T08:27:23.978Z",
    "ecs": {
      "version": "1.5.0"
    }
  },
  "fields": {
    "event.start": [
      "2020-10-01T08:27:22.474Z"
    ],
    "event.end": [
      "2020-10-01T08:27:23.978Z"
    ],
    "@timestamp": [
      "2020-10-01T08:27:23.978Z"
    ]
  }
}

Similar problem with another index - windowsdefender

{
  "_index": ".kibana-event-log-7.9.0-000002",
  "_type": "_doc",
  "_id": "L4ZL43QBf6VWBkBRV1S0",
  "_score": 1,
  "_source": {
    "event": {
      "provider": "alerting",
      "action": "execute",
      "start": "2020-10-01T08:33:17.647Z",
      "end": "2020-10-01T08:33:19.796Z",
      "duration": 2149000000,
      "outcome": "success"
    },
    "kibana": {
      "saved_objects": [
        {
          "rel": "primary",
          "type": "alert",
          "id": "caa921df-65f1-492c-9923-b24ef5271dc3"
        }
      ],
      "server_uuid": "db28e422-3b70-4b7d-afbd-1c94869e3914"
    },
    "message": "alert executed: siem.signals:caa921df-65f1-492c-9923-b24ef5271dc3: 'Virus Infected Machines (Quarantine Failure) - SNOW'",
    "@timestamp": "2020-10-01T08:33:19.796Z",
    "ecs": {
      "version": "1.5.0"
    }
  },
  "fields": {
    "event.start": [
      "2020-10-01T08:33:17.647Z"
    ],
    "event.end": [
      "2020-10-01T08:33:19.796Z"
    ],
    "@timestamp": [
      "2020-10-01T08:33:19.796Z"
    ]
  }
}