# Custom Rules not working **URL:** \ **Category:** SIEM\ **Tags:** elastic-stack-alerting\ **Created:** [December 11, 2020, 2:10pm UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421 "2020-12-11T14:10:34Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Yuriy\_Tsarenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuriy_tsarenko/32/72763_2.png) [@Yuriy\_Tsarenko](https://discuss.elastic.co/u/Yuriy_Tsarenko)\ **Post date:** [December 11, 2020, 2:10pm UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421/1 "2020-12-11T14:10:34Z")
Hello team! If possible, you could help us with setting up custom rules for SIEM. We had the following problem: The main idea is to create a rule according to which we would receive notifications when someone from the employee visits the Facebook site. We'd like to create our own signal rules and faces issues with the simplest one. But when creating a rule, it simply does not work out, please help figure it out. Adding exported detection rule and the json of the document from one of wathcquard-\* index ![rule_2](https://us1.discourse-cdn.com/elastic/original/3X/1/b/1b7bf680a1deae03d034b9da3485e754fb1d2976.png) ![rule_1](https://us1.discourse-cdn.com/elastic/original/3X/f/a/fae0811bd3aa85cba10770b63b934d255afaf698.png) ---
**Author:** ![ESamir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/esamir/32/77283_2.png) [@ESamir](https://discuss.elastic.co/u/ESamir)\ **Post date:** [December 11, 2020, 4:05pm UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421/2 "2020-12-11T16:05:30Z")
Hi How did you index the field dstname? are you using keyword or text ? can you share you mapping ? Regards ---
**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\ **Post date:** [December 14, 2020, 1:02pm UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421/3 "2020-12-14T13:02:14Z")
Hi @Yuriy_Tsarenko, welcome to our community! I want to ensure that you are aware of [Elastic Common Schema](https://www.elastic.co/what-is/ecs) (ECS). The Elastic SIEM/Security app, including its detection rules, signals, and detection alerts, _requires_ your data to be indexed in an ECS-compliant format. [ECS](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) is an open source, community-developed schema that specifies field names and Elasticsearch data types for each field, and provides descriptions and example usage. The easiest way to get your data in ECS-compliant format is to use an Elastic-supplied beat module, (e.g., [filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html) or Elastic Agent integration), which will ingest and index your data in an ECS-compliant format. Elastic provides a growing list of these integrations that you can find on our [Integrations page](https://www.elastic.co/integrations?solution=security). If you're using a custom data ingestion method (beat, Logstash, Ingest node pipeline), or one provided by a third-party, then you may need to convert your data so that it is in an ECS-compliant format before you can use the SIEM/security app. This can be done by creating your own beat/module, or your own Logstash configuration for each data source, which will convert your data to ECS during the ingestion process. [General guidelines](https://www.elastic.co/guide/en/ecs/current/ecs-guidelines.html) for creating ECS-compliant data: 1. Each indexed document (e.g., your log, event, etc.) MUST have the `@timestamp` field. 2. Your index mapping template must specify the [Elasticsearch field data type](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/mapping-types.html) for each field as defined by ECS. For example, your `@timestamp` field must be of the `date` field data type, etc.. This ensures that there will not be any mapping conflicts in your indices. 3. The original fields from your log/event SHOULD be copied/renamed/converted to the corresponding ECS-defined field name and data type. 4. Additional ECS fields, such as the [ECS Categorization fields](https://www.elastic.co/guide/en/ecs/current/ecs-category-field-values-reference.html) SHOULD be populated for each log/event, to allow proper inclusion of your data into dashboards and detection rules. A list of the specific ECS fields used by the SIEM/Security app is provided in this [reference](https://www.elastic.co/guide/en/security/current/siem-field-reference.html). I am guessing that your Elasticsearch index mapping for `host` may not be compliant with ECS. Your document is using `host` to hold an IP address, but ECS defines `host` as a field set object with multiple `host.*` fields defined [here](https://www.elastic.co/guide/en/ecs/current/ecs-host.html). Sorry for the information dump, but we've found that non-ECS-compliant data is a common root cause for users who experience problems getting their SIEM/Security app rules/signals to work. Please let us know if this is helpful. ---
**Author:** ![Yuriy\_Tsarenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuriy_tsarenko/32/72763_2.png) [@Yuriy\_Tsarenko](https://discuss.elastic.co/u/Yuriy_Tsarenko)\ **Post date:** [December 15, 2020, 11:33am UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421/4 "2020-12-15T11:33:45Z")
Hello and thank you for the response. We aware about ECS for SIEM working with the predefined rules. But what if we would like to use custom rules ? Following this manual [https://www.elastic.co/guide/en/security/current/rules-ui-create.html](https://www.elastic.co/guide/en/security/current/rules-ui-create.html) "Custom query: Query-based rule, which searches the defined indices and creates an alert when a document matches the rule’s query." We need simply run KQL query in specific index. So as I understand we do not need ECS for custom detection rules. What we're doing: - define what index we should use for search and add it to Kibana → Stack Management → Advanced Settings → securitysolution:defaultIndex (index has no relation to ECS) - go to discover and see if our KQL rule works - create a new detection rule with tested KQL query and enter the index pattern where we would like to run the query. - adding schedule, actions - save and enable rule Please correct me if there is a mistake. Thank you in advance ---
**Author:** ![borna\_talebi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/borna_talebi/32/76317_2.png) [@borna\_talebi](https://discuss.elastic.co/u/borna_talebi)\ **Post date:** [December 15, 2020, 11:44am UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421/5 "2020-12-15T11:44:56Z")
> [@Yuriy\_Tsarenko](#): > > But what if we would like to use custom rules ? You need to use ECS for your mapping when creating custom rules. I had a [similar problem](https://discuss.elastic.co/t/siem-rule-not-working-for-custom-query/253895) too. ---
**Author:** ![Yuriy\_Tsarenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuriy_tsarenko/32/72763_2.png) [@Yuriy\_Tsarenko](https://discuss.elastic.co/u/Yuriy_Tsarenko)\ **Post date:** [December 15, 2020, 2:27pm UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421/6 "2020-12-15T14:27:14Z")
Hello and mighty thank you for response. To be 100% sure I understood everything right please give me an answer for next: We have schema for fields naming and giving them some mapping (ECS). And SIEM works only with with index that follows ECS. In other words if we have some custom field with naming that is not described in ECS, we cannot use it for SIEM not only for predefined rules but entirely for any analytics by SIEM ? ---
**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\ **Post date:** [December 15, 2020, 2:56pm UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421/7 "2020-12-15T14:56:07Z")
Can you please share the rule and a sample document as json here instead of posting screenshoot, so anyone can try to reproduce the issue. ---
**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\ **Post date:** [December 16, 2020, 3:27pm UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421/8 "2020-12-16T15:27:01Z")
Hi @Yuriy_Tsarenko > We have schema for fields naming and giving them some mapping (ECS). And SIEM works only with with index that follows ECS. Correct. The Elastic SIEM/Security App _requires_ that all data must be ECS-compliant. The app works _best_ and _most-fully_ when the data has been fully converted to ECS format. ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/b/3be2a8aeaf59e94c2b3396241e9b28d603ad9a4a.jpeg) > In other words if we have some custom field with naming that is not described in ECS, we cannot use it for SIEM not only for predefined rules but entirely for any analytics by SIEM ? Not correct. ECS is a permissive schema. If your events have additional data that cannot be mapped to ECS, you can simply add them to your events, using custom field names, and your data can still be ECS-compliant. However, any custom fields you add _must not conflict_ with ECS-defined fields. AND if you use any ECS-defined fields, your Elasticsearch field data type mapping for them _must not conflict_ with those specified by ECS. From your screen shot above, it is clear that your data has not been mapped to ECS at all. Further it appears that your data is NOT ECS-compatible, as it may have a mapping for `host` that conflicts with ECS use of `host.*`, which would cause your rule not work. As suggested below, please feel free to attach an actual log sample (please ensure no private information is included) as well as the index mappings for the index into which your watchguard logs are being indexed. Finally, can you tell us what method you are using to ingest your data? e.g., Logstash? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 13, 2021, 3:27pm UTC](https://discuss.elastic.co/t/custom-rules-not-working/258421/9 "2021-01-13T15:27:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.