Hi,
I tried to create a ML rule for detecting signals when there is an anomaly in a ML job.
The ML job is clearly showing anomalies but the rule is not detecting signals.
Further, the rule is getting executed successfully and there is no failure history.
The logs are like this:
log [13:33:23.801] [debug][plugins][taskManager][taskManager] Running task alerting:siem.signals "ddac93a0-e3b2-11ea-a2e0-7fc0a0c89019"
log [13:33:23.847] [debug][plugins][plugins][siem][siem] [+] Starting Signal Rule execution name: "alj1" id: "24102dbc-80ab-4444-83ba-e82579bb039c" rule id: "c13f7c3e-aafe-4406-9141-7865e2fe1bda" signals index: ".siem-signals-default"
log [13:33:23.851] [debug][collector-set][plugins][usageCollection] Fetching data from kibana_stats collector
log [13:33:23.852] [debug][collector-set][plugins][usageCollection] Fetching data from kibana_settings collector
log [13:33:23.863] [debug][collector-set][plugins][usageCollection] not sending [kibana_settings] monitoring document because [undefined] is null or invalid.
log [13:33:23.867] [debug][kibana-monitoring][monitoring][monitoring][plugins] Uploading bulk stats payload to the local cluster
log [13:33:23.871] [debug][kibana-monitoring][monitoring][monitoring][plugins] Uploaded bulk stats payload to the local cluster
log [13:33:24.104] [info][plugins][plugins][siem][siem] Found 0 signals for notification. name: "alj1" id: "24102dbc-80ab-4444-83ba-e82579bb039c" rule id: "c13f7c3e-aafe-4406-9141-7865e2fe1bda" signals index: ".siem-signals-default"
log [13:33:24.104] [debug][plugins][plugins][siem][siem] [+] Signal Rule execution completed. name: "alj1" id: "24102dbc-80ab-4444-83ba-e82579bb039c" rule id: "c13f7c3e-aafe-4406-9141-7865e2fe1bda" signals index: ".siem-signals-default"
log [13:33:25.131] [debug][eventLog][plugins] writing to event log: {"index":".kibana-event-log-7.8.1","body":{"event":{"provider":"alerting","action":"execute","start":"2020-08-21T13:33:23.837Z","end":"2020-08-21T13:33:25.129Z","duration":1292000000,"outcome":"success"},"kibana":{"saved_objects":[{"rel":"primary","type":"alert","id":"24102dbc-80ab-4444-83ba-e82579bb039c"}],"server_uuid":"37fcb037-a040-42a1-bcb8-76b9b5ec8d45"},"message":"alert executed: siem.signals:24102dbc-80ab-4444-83ba-e82579bb039c: 'alj1'","@timestamp":"2020-08-21T13:33:25.129Z","ecs":{"version":"1.5.0"}}}
log [13:33:25.132] [debug][eventLog][plugins] esContext: callEs(index) calls: {"index":".kibana-event-log-7.8.1","body":{"event":{"provider":"alerting","action":"execute","start":"2020-08-21T13:33:23.837Z","end":"2020-08-21T13:33:25.129Z","duration":1292000000,"outcome":"success"},"kibana":{"saved_objects":[{"rel":"primary","type":"alert","id":"24102dbc-80ab-4444-83ba-e82579bb039c"}],"server_uuid":"37fcb037-a040-42a1-bcb8-76b9b5ec8d45"},"message":"alert executed: siem.signals:24102dbc-80ab-4444-83ba-e82579bb039c: 'alj1'","@timestamp":"2020-08-21T13:33:25.129Z","ecs":{"version":"1.5.0"}}}
log [13:33:25.458] [debug][eventLog][plugins] esContext: callEs(index) result: {"_index":".kibana-event-log-7.8.1-000001","_type":"_doc","_id":"VcM5EXQBjubZkO_PWVcQ","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":8990,"_primary_term":24}
log [13:33:25.460] [debug][eventLog][plugins] writing to event log complete
Additionally, the time field of the index used for ML job is not current time.
After detecting signals there is an action that need to be executed.
My primary goal is to get alerts when there is an anomaly in my index.
Will be happy if I get to know any other way for this process.
Thanks.