Detection Rule Error

[Anabella_Cristaldi (Anna)]

Hi,
I'm running the 7.9.2 version of the stack in a hot-warm cluster architecture.
When defining a very simple detection rule on my space called "siem" I get the following error intermittently. One execution succeed and one fails with:

    {"type":"log","@timestamp":"2020-10-21T14:16:06Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":6587,"message":"[-] search_after and bulk threw an error TypeError: Cannot read property 'some' of undefined name: \"Rogue AP Detection\" id: \"376e5caf-7fa0-4657-87b5-33ee249f9b3b\" rule id: \"b57a7041-d90f-4023-adf4-09e19182dcea\" signals index: \".siem-signals-siem\""}
    {"type":"log","@timestamp":"2020-10-21T14:16:06Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":6587,"message":"Bulk Indexing of signals failed. Check logs for further details. name: \"Rogue AP Detection\" id: \"376e5caf-7fa0-4657-87b5-33ee249f9b3b\" rule id: \"b57a7041-d90f-4023-adf4-09e19182dcea\" signals index: \".siem-signals-siem\""}
    {"type":"log","@timestamp":"2020-10-21T14:36:18Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":6587,"message":"[-] search_after and bulk threw an error TypeError: Cannot read property 'some' of undefined name: \"Rogue AP Detection\" id: \"376e5caf-7fa0-4657-87b5-33ee249f9b3b\" rule id: \"b57a7041-d90f-4023-adf4-09e19182dcea\" signals index: \".siem-signals-siem\""}
    {"type":"log","@timestamp":"2020-10-21T14:36:18Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":6587,"message":"Bulk Indexing of signals failed. Check logs for further details. name: \"Rogue AP Detection\" id: \"376e5caf-7fa0-4657-87b5-33ee249f9b3b\" rule id: \"b57a7041-d90f-4023-adf4-09e19182dcea\" signals index: \".siem-signals-siem\""}

Thank you
Regards
Ana

[Frank_Hassanabad]

Hi @Anabella_Cristaldi,

Is it ok for you to share the contents of that rule by exporting it and posting it here?

It looks like from the errors you have a space called siem as well? Something like this?

Any other information such as what does your source index mapping look like or sample data from your source index(es) you're using would be helpful too.

I just did a quick sanity check with auditbeat, Elastic cloud hot-warm cluster architecture

and then created a space with the id of siem and didn't see anything obvious that pops out.

[Anabella_Cristaldi (Anna)]

Hi @Frank_Hassanabad,
Thank you for your reply.
Here the rule export

{"author":[],"actions":[],"created_at":"2020-10-21T14:10:58.170Z","updated_at":"2020-10-21T14:58:36.538Z","created_by":"acristal","description":"Rogue AP Detection","enabled":true,"false_positives":[],"filters":[],"from":"now-3900s","id":"376e5caf-7fa0-4657-87b5-33ee249f9b3b","immutable":false,"index":["forti-logs*"],"interval":"5m","rule_id":"b57a7041-d90f-4023-adf4-09e19182dcea","language":"kuery","license":"","output_index":".siem-signals-siem","max_signals":100,"risk_score":74,"risk_score_mapping":[],"name":"Rogue AP Detection","query":"logid: \"0104043563\"","references":[],"meta":{"from":"60m","kibana_siem_app_url":"https://192.168.1.93:5601/s/siem/app/security"},"severity":"critical","severity_mapping":[],"updated_by":"acristal","tags":[],"to":"now","type":"query","threat":[],"throttle":"no_actions","version":2,"exceptions_list":[]}
{"exported_count":1,"missing_rules":[],"missing_rules_count":0} 

Also my index mapping
Index Mapping

It seems that the are something it the space SIEM that broke the rule. I did a few tests.
Here the results in order to provide more information:

• I've create another space called IGOR, export ALL objects from SIEM and imported into IGOR.
  Then I create the detection rule, same error.

• I've created a 3rd space called Test, export ONLY Index-patterns, dashboards, visualizations, maps, searches and lens from SIEM into Test.
  Then I've create the same rule in the new one and I do not have the error (although It is not working as expected. Same problem that Detections with custom query)

• I have a separate ELK instance running with an space called SIEM. I create the rule and after a few successful execution, have the same error
  

  

My conclusion is that the error is somehow related to the name space "SIEM" event when created a new Space with a different name and import ALL objects into the new space, some configuration could remain with the siem name and broke the rule.

Thank you and sorry for my long reply
Regards
Anna

[Frank_Hassanabad]

I loaded up your mapping and didn't see any obvious conflicts and your rule looks ok. Nothing odd about it.

But your current UI error message indicates that if you have access to your Kibana logs you should have more information on the error message if you're running on prem which should point to the root of the problem. We are bubbling up those errors in the next upcoming release as you probably saw on my other forum post.

Is it ok to send me a data test sample from that index (not real data though, just a test sample) as well?

[Anabella_Cristaldi (Anna)]

Hi @Frank_Hassanabad,

The sample data and mappings are the same posted in Detections with custom query

Regarding to kibana logs, the error that appear is:

{"type":"log","@timestamp":"2020-10-21T14:16:06Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":6587,"message":"[-] search_after and bulk threw an error TypeError: Cannot read property 'some' of undefined name: \"Rogue AP Detection\" id: \"376e5caf-7fa0-4657-87b5-33ee249f9b3b\" rule id: \"b57a7041-d90f-4023-adf4-09e19182dcea\" signals index: \".siem-signals-siem\""}

But I still believe that is something related to the space name (siem). In the new space (temp) I did not have any errors.

Thank you
Regards
Anna

[Frank_Hassanabad]

You could have more than one error going on. I did post that I found a mapping conflict between your custom one and the ECS signals mapping:

Detections with custom query

Hi @Anabella_Cristaldi, Appreciate the sample thank you! I do not see why is working for threshold but not for custom query (Does they have a different mechanism for quering)? Yes, they do have different mechanisms. The threshold one is an aggregation and it does not fill in all the values when it creates a signal. So, some good news is that in the soon to be released 7.10.0 where we improved error handling you will begin to see errors on that rule where before you were not. I just …

I would clean up the mappings around host and double check to ensure there aren't other conflicts and then from there we can see if you still have an issue on this one with the space name.