Rules don't trigger and preview window is empty

[Alex_Bailey]

Working with Elastic 8.1 stack and I can't seem to get rules to work for me no matter what I try.

I have a rule defined as so:

When I run the preview, this brings back no results:

The request generated in this query window appears to be:

{
  "aggregations": {
    "preview": {
      "terms": {
        "field": "event.category",
        "missing": "All others",
        "order": {
          "_count": "desc"
        },
        "size": 10
      },
      "aggs": {
        "preview": {
          "date_histogram": {
            "field": "signal.original_time",
            "fixed_interval": "112500ms",
            "min_doc_count": 0,
            "extended_bounds": {
              "min": 1647788691195,
              "max": 1647792291195
            }
          }
        }
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [],
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "match": {
                        "kibana.alert.rule.uuid": "52f47123-16e4-45a8-a552-df4f236d1baf"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ],
            "should": [],
            "must_not": []
          }
        },
        {
          "range": {
            "signal.original_time": {
              "gte": "2022-03-20T15:04:51.195Z",
              "lte": "2022-03-20T16:04:51.195Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ]
    }
  },
  "size": 0
}

The response is:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": 0,
    "hits": []
  }
}

I have definitely got a document in the index which should match what I entered:

For testing purposes, I have both @timestamp and event.ingested as the same timestamp.

The document is as follows:

{
  "_index": ".ds-logs-aws-cloudtrail-2022.03.20-000001",
  "_id": "2xXlp38Bqw_w_LoT8Bul",
  "_version": 1,
  "_score": 1,
  "_source": {
    "log_file": "AWSLogs/************/CloudTrail/us-east-1/2022/03/20/************_CloudTrail_us-east-1_20220320T1515Z_************.json.gz",
    "event_version": "1.08",
    "request_id": "************",
    "event_type": "AwsApiCall",
    "management_event": true,
    "read_only": false,
    "recipient_account_id": "************",
    "event_category": "Management",
    "session_credential_from_console": "true",
    "user_identity": {
      "type": "AssumedRole",
      "principal_id": "************:************",
      "arn": "arn:aws:sts::************:assumed-role/AWSReservedSSO_AWSAdministratorAccess_************/************",
      "account_id": "************",
      "access_key_id": "************",
      "session_context": {},
      "session_issuer": {}
    },
    "web_id_federation_data": {},
    "ecs": {
      "version": "8.1.0"
    },
    "error": {},
    "cloud": {
      "provider": "aws",
      "region": "us-east-1",
      "account": {
        "id": "************"
      }
    },
    "tls": {},
    "user_agent": {
      "original": "AWS Internal"
    },
    "client": {
      "ip": "AWS Internal"
    },
    "event": {
      "id": "************",
      "kind": "event",
      "category": [
        "iam"
      ],
      "action": "CreateGroup",
      "type": [
        "info"
      ],
      "ingested": "2022-03-20T15:10:41Z",
      "provider": "iam.amazonaws.com",
      "dataset": "aws.cloudtrail",
      "outcome": "success"
    },
    "user": {
      "id": "arn:aws:sts::************:assumed-role/AWSReservedSSO_AWSAdministratorAccess_************/************"
    },
    "@timestamp": "2022-03-20T15:10:41Z"
  },
  "fields": {
    "session_credential_from_console": [
      true
    ],
    "event.category": [
      "iam"
    ],
    "user_identity.account_id": [
      "************"
    ],
    "user_identity.principal_id": [
      "************:************"
    ],
    "event_version": [
      "1.08"
    ],
    "event_type": [
      "AwsApiCall"
    ],
    "read_only": [
      false
    ],
    "log_file": [
      "AWSLogs/************/CloudTrail/us-east-1/2022/03/20/************_CloudTrail_us-east-1_20220320T1515Z_************.json.gz"
    ],
    "user_identity.access_key_id": [
      "************"
    ],
    "event.kind": [
      "event"
    ],
    "event.outcome": [
      "success"
    ],
    "user_agent.original": [
      "AWS Internal"
    ],
    "cloud.region": [
      "us-east-1"
    ],
    "user.id": [
      "arn:aws:sts::************:assumed-role/AWSReservedSSO_AWSAdministratorAccess_************/************"
    ],
    "user_identity.type": [
      "AssumedRole"
    ],
    "client.ip": [
      "AWS Internal"
    ],
    "recipient_account_id": [
      "************"
    ],
    "event_category": [
      "Management"
    ],
    "cloud.provider": [
      "aws"
    ],
    "event.provider": [
      "iam.amazonaws.com"
    ],
    "event.ingested": [
      "2022-03-20T15:10:41.000Z"
    ],
    "event.action": [
      "CreateGroup"
    ],
    "@timestamp": [
      "2022-03-20T15:10:41.000Z"
    ],
    "ecs.version": [
      "8.1.0"
    ],
    "cloud.account.id": [
      "************"
    ],
    "event.type": [
      "info"
    ],
    "user_identity.arn": [
      "arn:aws:sts::************:assumed-role/AWSReservedSSO_AWSAdministratorAccess_************/************"
    ],
    "event.id": [
      "************"
    ],
    "request_id": [
      "************"
    ],
    "management_event": [
      true
    ],
    "event.dataset": [
      "aws.cloudtrail"
    ]
  }
}

Can anyone point me in the direction of why this data isn't showing in the preview window?

Edit: I should mention that there are no errors on the rule page and no failure history. The rule doesn't trigger or show results in the preview either.

Edit 2: Something I do find strange is the signal.original_time in the query generated for the preview. This isn't something I'm setting anywhere so why is Elastic adding it? What is its purpose?

Edit 3: Updated screenshot.

Edit 4: Might be worth mentioning that even if I do the rule filter as @timestamp : *, the preview is still empty. It's still empty if I set the index pattern to *.

Edit 5: I should mention this is a datastream and I've tried with the index pattern .ds-logs-aws-cloudtrail-*. Events do show on the security overview as shown below:

[Mike_Paquette]

Hi @Alex_Bailey , welcome to our community. We're glad you are trying out Elastic Security and we regret that you are having trouble getting your alerts to show up in the Rule's quick query preview.

I am not an expert on the rule query preview implementation, so we may need to get some additional help, but I have two quick observations that may or may not be relevant:

Time alignment: The @timestamp in the document you shared is 2022-03-20T15:10:41Z but the screenshot of Discover shows a time period from 17:56 to 18:56. Are we sure that there are matching documents during the "Last hour" when you ran the rule Quick query preview? Did you try running the preview over the "Last day" ?

Rule Query: I noticed that your rule users only filters. I think this should work, but as a test, have you tried using the query in the Custom query Search bar? Similar to the KQL query you entered in Discover?

Thanks,
Mike P.

[Alex_Bailey]

@Mike_Paquette

Rule Query: I noticed that your rule users only filters. I think this should work, but as a test, have you tried using the query in the Custom query Search bar? Similar to the KQL query you entered in Discover?

Hi Mike,

I've just tried this again and unfortunately still no luck. I had to update one of the images hence the strange timestamp.

Below are some updated images:

Discover last hour:

Custom Rule (last hour):

Custom Rule (last day):

I've tried all the different preview options and I've tried KQL as well as the filters. Unfortunately neither seems to help.

[vitaliidm (Vitalii Dmyterko)]

Hi @Alex_Bailey. Sorry to hear you encountered issue with rules preview.

I've tried to reproduce the issue with the data you kindly attached to your post.
And I can see the issue is with this field

    "client": {
      "ip": "AWS Internal"
    },

When running preview rule on index with this object, I was able to see following error

[2022-03-21T15:48:22.801+00:00][ERROR][plugins.ruleRegistry] ResponseError: {"took":0,"errors":true,"items":[{"create":{"_index":".internal.preview.alerts-security.alerts-default-000001","_id":"903fcb1bcc76909441a75740784a0abfc5d08235a95bcc7d5e048490a1193168","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse field [client.ip] of type [ip] in document with id '903fcb1bcc76909441a75740784a0abfc5d08235a95bcc7d5e048490a1193168'. Preview of field's value: 'AWS Internal'","caused_by":{"type":"illegal_argument_exception","reason":"'AWS Internal' is not an IP string literal."}}}}]}

During rule preview, data is gets copied into this index, which has mapping for client.ip as as ip type.
Here list of ECS compliant data format Client Fields | Elastic Common Schema (ECS) Reference [8.1] | Elastic

After removing this field from data, I was able to see preview results

[Alex_Bailey]

I've just tried this on a new document (with the correct IP):

{
  "_index": ".ds-logs-aws-cloudtrail-2022.03.21-000001",
  "_id": "pRUjsX8Bqw_w_LoTlr8F",
  "_version": 1,
  "_score": 1,
  "_source": {
    "agent": {
      "type": "platform-logging"
    },
    "log": {
      "file": {
        "path": "AWSLogs/*******/CloudTrail/us-east-1/2022/03/22/*******_CloudTrail_us-east-1_20220322T1015Z_c2KfftAUDHzFHMVt.json.gz"
      },
      "level": "info",
      "logger": "cloudtrail"
    },
    "error": {},
    "cloud": {
      "provider": "aws",
      "service": {
        "name": "iam.amazonaws.com"
      },
      "region": "us-east-1",
      "account": {
        "id": "********"
      }
    },
    "@timestamp": "2022-03-22T10:14:26Z",
    "ecs": {
      "version": "8.0.0"
    },
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "aws.cloudtrail"
    },
    "client": {
      "ip": "0.0.0.0"
    },
    "tls": {},
    "api": {},
    "event": {
      "original": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"AssumedRole\",\"principalId\":\"*******:abailey\",\"arn\":\"arn:aws:sts::*******:assumed-role/AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda/abailey\",\"accountId\":\"*******\",\"accessKeyId\":\"*******\",\"sessionContext\":{\"sessionIssuer\":{\"type\":\"Role\",\"principalId\":\"*******\",\"arn\":\"arn:aws:iam::*******:role/aws-reserved/sso.amazonaws.com/eu-west-1/AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda\",\"accountId\":\"787743944430\",\"userName\":\"AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda\"},\"webIdFederationData\":{},\"attributes\":{\"creationDate\":\"2022-03-22T10:13:55Z\",\"mfaAuthenticated\":\"false\"}}},\"eventTime\":\"2022-03-22T10:14:26Z\",\"eventSource\":\"iam.amazonaws.com\",\"eventName\":\"CreateGroup\",\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"AWS Internal\",\"userAgent\":\"AWS Internal\",\"requestParameters\":{\"groupName\":\"test-alert-group\"},\"responseElements\":{\"group\":{\"path\":\"/\",\"groupName\":\"test-alert-group\",\"groupId\":\"*******\",\"arn\":\"arn:aws:iam::*******:group/test-alert-group\",\"createDate\":\"Mar 22, 2022 10:14:26 AM\"}},\"requestID\":\"37a01acd-a715-41bf-8dcd-786f57a8f8e1\",\"eventID\":\"82683467-3638-40b1-bab3-2d95ab4707af\",\"readOnly\":false,\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"*******\",\"eventCategory\":\"Management\",\"sessionCredentialFromConsole\":\"true\"}",
      "kind": "event",
      "module": "aws",
      "type": [
        "info"
      ],
      "version": "1.08",
      "ingested": "2022-03-22T10:19:33.765077006Z",
      "provider": "iam.amazonaws.com",
      "management": true,
      "read_only": false,
      "action": "CreateGroup",
      "id": "82683467-3638-40b1-bab3-2d95ab4707af",
      "category": [
        "iam"
      ],
      "dataset": "aws.cloudtrail",
      "outcome": "success"
    },
    "user": {
      "access_key_id": "*******",
      "session_issuer": {},
      "account_id": "********",
      "session_context": {},
      "id": "arn:aws:sts::*******:assumed-role/AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda/abailey",
      "type": "AssumedRole",
      "arn": "arn:aws:sts::*******:assumed-role/AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda/abailey",
      "principal_id": "*******:abailey"
    }
  },
  "fields": {
    "event.category": [
      "iam"
    ],
    "user.principal_id": [
      "*******:abailey"
    ],
    "log.logger": [
      "cloudtrail"
    ],
    "event.version": [
      1.08
    ],
    "agent.type": [
      "platform-logging"
    ],
    "event.module": [
      "aws"
    ],
    "log.level": [
      "info"
    ],
    "user.account_id": [
      "********"
    ],
    "event.kind": [
      "event"
    ],
    "event.management": [
      true
    ],
    "user.access_key_id": [
      "********"
    ],
    "event.outcome": [
      "success"
    ],
    "agent.type.keyword": [
      "platform-logging"
    ],
    "event.original": [
      "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"AssumedRole\",\"principalId\":\"*******:abailey\",\"arn\":\"arn:aws:sts::*******:assumed-role/AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda/abailey\",\"accountId\":\"*******\",\"accessKeyId\":\"*******\",\"sessionContext\":{\"sessionIssuer\":{\"type\":\"Role\",\"principalId\":\"*******\",\"arn\":\"arn:aws:iam::*******:role/aws-reserved/sso.amazonaws.com/eu-west-1/AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda\",\"accountId\":\"*******\",\"userName\":\"AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda\"},\"webIdFederationData\":{},\"attributes\":{\"creationDate\":\"2022-03-22T10:13:55Z\",\"mfaAuthenticated\":\"false\"}}},\"eventTime\":\"2022-03-22T10:14:26Z\",\"eventSource\":\"iam.amazonaws.com\",\"eventName\":\"CreateGroup\",\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"AWS Internal\",\"userAgent\":\"AWS Internal\",\"requestParameters\":{\"groupName\":\"test-alert-group\"},\"responseElements\":{\"group\":{\"path\":\"/\",\"groupName\":\"test-alert-group\",\"groupId\":\"*******\",\"arn\":\"arn:aws:iam::*******:group/test-alert-group\",\"createDate\":\"Mar 22, 2022 10:14:26 AM\"}},\"requestID\":\"37a01acd-a715-41bf-8dcd-786f57a8f8e1\",\"eventID\":\"82683467-3638-40b1-bab3-2d95ab4707af\",\"readOnly\":false,\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"********\",\"eventCategory\":\"Management\",\"sessionCredentialFromConsole\":\"true\"}"
    ],
    "cloud.region": [
      "us-east-1"
    ],
    "event.read_only": [
      false
    ],
    "user.id": [
      "arn:aws:sts::*******:assumed-role/AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda/abailey"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "client.ip": [
      "0.0.0.0"
    ],
    "data_stream.type": [
      "logs"
    ],
    "cloud.provider": [
      "aws"
    ],
    "event.provider": [
      "iam.amazonaws.com"
    ],
    "event.ingested": [
      "2022-03-22T10:19:33.765Z"
    ],
    "event.action": [
      "CreateGroup"
    ],
    "@timestamp": [
      "2022-03-22T10:14:26.000Z"
    ],
    "cloud.service.name": [
      "iam.amazonaws.com"
    ],
    "cloud.account.id": [
      "********"
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "user.arn": [
      "arn:aws:sts::*******:assumed-role/AWSReservedSSO_AWSAdministratorAccess_3b59e66dd6d68dda/abailey"
    ],
    "log.file.path": [
      "AWSLogs/*******/CloudTrail/us-east-1/2022/03/22/*******_CloudTrail_us-east-1_20220322T1015Z_c2KfftAUDHzFHMVt.json.gz"
    ],
    "data_stream.dataset": [
      "aws.cloudtrail"
    ],
    "event.type": [
      "info"
    ],
    "user.type": [
      "AssumedRole"
    ],
    "event.id": [
      "82683467-3638-40b1-bab3-2d95ab4707af"
    ],
    "event.dataset": [
      "aws.cloudtrail"
    ]
  }
}

In a brand new cluster with no changes at all, this works and shows in the preview. In the existing cluster, this does not work with the same document.

We have set up components in the not-working cluster with the ECS field mapping as below:

{
  "_routing": {
    "required": false
  },
  "numeric_detection": false,
  "dynamic_date_formats": [
    "basic_date_time_no_millis",
    "date_time_no_millis"
  ],
  "dynamic": true,
  "_source": {
    "excludes": [],
    "includes": [],
    "enabled": true
  },
  "date_detection": true,
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "message": {
      "type": "text"
    },
    "labels": {
      "type": "keyword"
    },
    "tags": {
      "type": "keyword"
    }
  }
}

{
  "properties": {
    "client": {
      "dynamic": false,
      "type": "object",
      "enabled": true,
      "properties": {
        "nat": {
          "type": "object",
          "properties": {
            "port": {
              "type": "long"
            },
            "ip": {
              "type": "ip"
            }
          }
        },
        "registered_domain": {
          "type": "keyword"
        },
        "address": {
          "type": "keyword"
        },
        "port": {
          "type": "long"
        },
        "top_level_domain": {
          "type": "keyword"
        },
        "bytes": {
          "type": "long"
        },
        "domain": {
          "type": "keyword"
        },
        "ip": {
          "type": "ip"
        },
        "subdomain": {
          "type": "keyword"
        },
        "mac": {
          "type": "keyword"
        },
        "packets": {
          "type": "long"
        }
      }
    }
  }
}

{
  "dynamic_templates": [],
  "properties": {
    "cloud": {
      "dynamic": false,
      "type": "object",
      "enabled": true,
      "properties": {
        "availability_zone": {
          "type": "keyword"
        },
        "instance": {
          "type": "object",
          "properties": {
            "name": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            }
          }
        },
        "macine": {
          "type": "object",
          "properties": {
            "type": {
              "type": "keyword"
            }
          }
        },
        "provider": {
          "type": "keyword"
        },
        "service": {
          "type": "object",
          "properties": {
            "name": {
              "type": "keyword"
            }
          }
        },
        "project": {
          "type": "object",
          "properties": {
            "name": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            }
          }
        },
        "region": {
          "type": "keyword"
        },
        "account": {
          "dynamic": false,
          "type": "object",
          "enabled": true,
          "properties": {
            "name": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

{
  "dynamic_templates": [],
  "properties": {
    "data_stream": {
      "type": "object",
      "properties": {
        "namespace": {
          "type": "keyword"
        },
        "type": {
          "type": "keyword"
        },
        "dataset": {
          "type": "keyword"
        }
      }
    }
  }
}

{
  "properties": {
    "ecs": {
      "type": "object",
      "properties": {
        "version": {
          "type": "keyword"
        }
      }
    }
  }
}

{
  "properties": {
    "error": {
      "dynamic": false,
      "type": "object",
      "enabled": true,
      "properties": {
        "code": {
          "type": "keyword"
        },
        "id": {
          "type": "keyword"
        },
        "stack_trace": {
          "type": "text"
        },
        "message": {
          "type": "text"
        },
        "type": {
          "type": "keyword"
        }
      }
    }
  }
}

{
  "properties": {
    "event": {
      "dynamic": true,
      "type": "object",
      "enabled": true,
      "properties": {
        "reason": {
          "type": "keyword"
        },
        "code": {
          "type": "keyword"
        },
        "timezone": {
          "type": "keyword"
        },
        "type": {
          "type": "keyword"
        },
        "agent_id_status": {
          "type": "keyword"
        },
        "duration": {
          "type": "long"
        },
        "reference": {
          "type": "keyword"
        },
        "ingested": {
          "type": "date"
        },
        "provider": {
          "type": "keyword"
        },
        "action": {
          "type": "keyword"
        },
        "end": {
          "type": "date"
        },
        "id": {
          "type": "text"
        },
        "outcome": {
          "type": "keyword"
        },
        "severity": {
          "type": "long"
        },
        "original": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": false,
          "store": false,
          "type": "keyword",
          "split_queries_on_whitespace": false,
          "doc_values": false
        },
        "risk_score": {
          "type": "float"
        },
        "kind": {
          "type": "keyword"
        },
        "created": {
          "type": "date"
        },
        "module": {
          "type": "keyword"
        },
        "start": {
          "type": "date"
        },
        "url": {
          "type": "keyword"
        },
        "sequence": {
          "type": "long"
        },
        "risk_score_norm": {
          "type": "float"
        },
        "category": {
          "type": "keyword"
        },
        "dataset": {
          "type": "keyword"
        },
        "hash": {
          "type": "keyword"
        }
      }
    }
  }
}

{
  "properties": {
    "log": {
      "dynamic": true,
      "type": "object",
      "enabled": true,
      "properties": {
        "file": {
          "dynamic": true,
          "type": "object",
          "enabled": true,
          "properties": {
            "path": {
              "type": "keyword"
            }
          }
        },
        "level": {
          "type": "keyword"
        },
        "logger": {
          "type": "keyword"
        },
        "origin": {
          "type": "object",
          "properties": {
            "file": {
              "type": "object",
              "properties": {
                "line": {
                  "type": "long"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "function": {
              "eager_global_ordinals": false,
              "norms": false,
              "index": true,
              "store": false,
              "type": "keyword",
              "index_options": "docs",
              "split_queries_on_whitespace": false,
              "doc_values": true
            }
          }
        },
        "syslog": {
          "type": "object",
          "properties": {
            "severity": {
              "type": "object",
              "properties": {
                "code": {
                  "type": "long"
                },
                "name": {
                  "type": "keyword"
                }
              }
            },
            "priority": {
              "type": "long"
            },
            "facility": {
              "type": "object",
              "properties": {
                "code": {
                  "type": "long"
                },
                "name": {
                  "type": "keyword"
                }
              }
            }
          }
        }
      }
    }
  }
}

{
  "properties": {
    "tls": {
      "dynamic": false,
      "type": "object",
      "enabled": true,
      "properties": {
        "cipher": {
          "type": "keyword"
        },
        "established": {
          "type": "boolean"
        },
        "server": {
          "type": "object",
          "properties": {
            "not_after": {
              "type": "date"
            },
            "ja3s": {
              "type": "keyword"
            },
            "not_before": {
              "type": "date"
            },
            "subject": {
              "type": "keyword"
            },
            "certificate": {
              "type": "keyword"
            },
            "version": {
              "type": "keyword"
            },
            "certificate_chain": {
              "type": "keyword"
            },
            "hash": {
              "type": "object",
              "properties": {
                "sha1": {
                  "type": "keyword"
                },
                "sha256": {
                  "type": "keyword"
                },
                "md5": {
                  "type": "keyword"
                }
              }
            },
            "issuer": {
              "type": "keyword"
            },
            "version_protocol": {
              "type": "keyword"
            }
          }
        },
        "curve": {
          "type": "keyword"
        },
        "client": {
          "type": "object",
          "properties": {
            "not_after": {
              "type": "date"
            },
            "server_name": {
              "type": "keyword"
            },
            "not_before": {
              "type": "date"
            },
            "subject": {
              "type": "keyword"
            },
            "supported_ciphers": {
              "type": "keyword"
            },
            "certificate": {
              "type": "keyword"
            },
            "ja3": {
              "type": "keyword"
            },
            "certificate_chain": {
              "type": "keyword"
            },
            "hash": {
              "type": "object",
              "properties": {
                "sha1": {
                  "type": "keyword"
                },
                "sha256": {
                  "type": "keyword"
                },
                "md5": {
                  "type": "keyword"
                }
              }
            },
            "issuer": {
              "type": "keyword"
            }
          }
        },
        "next_protocol": {
          "type": "keyword"
        },
        "resumed": {
          "type": "boolean"
        },
        "version": {
          "type": "keyword"
        }
      }
    }
  }
}

{
  "_routing": {
    "required": false
  },
  "numeric_detection": false,
  "dynamic_date_formats": [
    "strict_date_optional_time",
    "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
  ],
  "dynamic": true,
  "_source": {
    "excludes": [],
    "includes": [],
    "enabled": true
  },
  "date_detection": true,
  "properties": {
    "user": {
      "dynamic": false,
      "type": "object",
      "enabled": true,
      "properties": {
        "full_name": {
          "type": "keyword"
        },
        "domain": {
          "type": "keyword"
        },
        "roles": {
          "type": "keyword"
        },
        "name": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "fields": {
            "text": {
              "type": "text"
            }
          },
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "id": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "email": {
          "type": "keyword"
        },
        "hash": {
          "type": "keyword"
        }
      }
    }
  }
}

{
  "properties": {
    "user_agent": {
      "dynamic": false,
      "type": "object",
      "enabled": true,
      "properties": {
        "original": {
          "eager_global_ordinals": false,
          "norms": false,
          "index": true,
          "store": false,
          "type": "keyword",
          "index_options": "docs",
          "split_queries_on_whitespace": false,
          "doc_values": true
        },
        "name": {
          "type": "keyword"
        },
        "device": {
          "type": "object",
          "properties": {
            "name": {
              "type": "keyword"
            }
          }
        },
        "version": {
          "type": "keyword"
        }
      }
    }
  }
}

As far as I can see, these all conform to the ECS schema. Is there a tool that we can use to check is mapping is causing a problem generating the preview?

[Pedro_Jaramillo]

Hi @Alex_Bailey, could you please verify that the proper rule preview index privileges are assigned to the user that is trying to use the Rule Preview feature, as described in this section of the user docs?

If the user does not have read privileges for the .preview.alerts-security.alerts-<space-id> index, the Rule Preview may fail to display preview results.

[Alex_Bailey]

Hi Pedro,

It was an admin user and had access to everything.

Yesterday we created a new cluster and set up everything from scratch. Everything is working in the new cluster.

In the old cluster, I think some of the 'default' objects had been deleted. For example, the .preview.alerts-security.alerts-<space-id> didn't exist. We tried to remedy this by creating it again but there were still issues. Due to growing concerns that more important indexes/template/etc had been deleted, we decided the best course of action was to create a cluster from scratch.

Thanks for the help. I'm sorry I can't provide more details on what was/wasn't deleted.