Detections with custom query

Frank_Hassanabad · October 27, 2020, 5:00am

Appreciate the sample thank you!

I do not see why is working for threshold but not for custom query (Does they have a different mechanism for quering)?

Yes, they do have different mechanisms. The threshold one is an aggregation and it does not fill in all the values when it creates a signal.

So, some good news is that in the soon to be released 7.10.0 where we improved error handling you will begin to see errors on that rule where before you were not. I just test ran that sample document off of Kibana master and here is the error:

It's pointing to your data set at host:

"host" : "srv",

Which has a conflict with the signal mapping. host has to be an object with inner objects/attributes as outlined here:

Once you fix that and re-index your data it should work. If it doesn't we can look at your mapping and data again. When the soon to be released 7.10.0 ships you will be able to see these error messages so getting these problems fixed sooner will be easier.

You can see the signals mapping here if it helps to find conflicts:

github.com

elastic/kibana/blob/main/x-pack/plugins/security_solution/server/lib/detection_engine/routes/index/ecs_mapping.json#L985


      
              "type": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "virtual_address": {
                "type": "long"
              },
              "virtual_size": {
                "type": "long"
              }
            },
            "type": "nested"
          },
          "segments": {
            "properties": {
              "sections": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "type": {
                "ignore_above": 1024,

We mostly use ECS tooling fwiw:

github.com

elastic/ecs/blob/main/USAGE.md

# ECS Tooling Usage

In addition to the published schema and artifacts, the ECS repo also contains tools to generate artifacts based on the current published and custom schemas.

You may be asking if ECS is a specification for storing event data, where does the ECS tooling fit into the picture?  As users implement ECS into their Elastic stack, common questions arise:

* ECS has too many fields. Users don't want to generate mappings for fields they don't plan on using soon.
* Users want to adopt ECS but also want to painlessly maintain their own custom field mappings alongside ECS.

Users can use the ECS tools to tackle both problems. What artifacts are relevant will also vary based on need. Many users will find the Elasticsearch templates most useful, but Beats
contributors will instead find the Beats-formatted YAML field definition files valuable. By maintaining only their customizations and use the tools provided by ECS, they can generate
relevant artifacts for their unique set of data sources.

**NOTE** - These tools and their functionality are considered experimental.

## Table of Contents

- [TLDR Example](#tldr-example)
- [Terminology](#terminology)
- [Setup and Install](#setup-and-install)

This file has been truncated. show original

and/or look at their generated outputs:
https://github.com/elastic/ecs/blob/master/generated/elasticsearch/7/template.json#L1071

to try and stay compliant and update along the way.