Our ML job stops execution with an exception: EmptyDataCountException: null

Elastic Security SIEM
1

Trying to run std. the various ML jobs for WIndows. Only two seems to keep running other stops again with exceptions like below. Wondering if exception happens maybe because we've still got to few sampled data yet?

TIA

[2019-12-17T15:15:37,055][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [d1r2n14] Opening job [windows_anomalous_path_activity_ecs]
[2019-12-17T15:15:37,062][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [d1r2n14] [windows_anomalous_path_activity_ecs] Loading model snapshot [1576582315] with latest_record_timestamp [N/A], job latest_record_timestamp [N/A]
[2019-12-17T15:15:37,063][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcessFactory] [d1r2n14] Restoring quantiles for job 'windows_anomalous_path_activity_ecs'
[2019-12-17T15:15:37,124][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [d1r2n14] [windows_anomalous_path_activity_ecs] [autodetect/26826] [CResourceMonitor.cc@70] Setting model memory limit to 256 MB
[2019-12-17T15:15:37,145][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [d1r2n14] [windows_anomalous_path_activity_ecs] [autodetect/26826] [CAnomalyJob.cc@839] Processing is already complete to time 1576582200
[2019-12-17T15:15:37,243][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [d1r2n14] Successfully set job state to [opened] for job [windows_anomalous_path_activity_ecs]
[2019-12-17T16:31:55,091][WARN ][o.e.x.m.d.DatafeedManager] [d1r2n14] Datafeed for [windows_anomalous_path_activity_ecs] has seen no data in [10] attempts, and never seen any data previously, so stopping...
[2019-12-17T16:31:55,091][INFO ][o.e.x.m.d.DatafeedManager] [d1r2n14] [no_data] attempt to stop datafeed [datafeed-windows_anomalous_path_activity_ecs] for job [windows_anomalous_path_activity_ecs]
[2019-12-17T16:31:55,091][INFO ][o.e.x.m.d.DatafeedManager] [d1r2n14] [no_data] try lock [20s] to stop datafeed [datafeed-windows_anomalous_path_activity_ecs] for job [windows_anomalous_path_activity_ecs]...
[2019-12-17T16:31:55,091][INFO ][o.e.x.m.d.DatafeedManager] [d1r2n14] [no_data] stopping datafeed [datafeed-windows_anomalous_path_activity_ecs] for job [windows_anomalous_path_activity_ecs], acquired [true]...
[2019-12-17T16:31:55,109][WARN ][o.e.p.AllocatedPersistentTask] [d1r2n14] task datafeed-datafeed-windows_anomalous_path_activity_ecs failed with an exception
org.elasticsearch.xpack.ml.datafeed.DatafeedJob$EmptyDataCountException: null
        at org.elasticsearch.xpack.ml.datafeed.DatafeedJob.run(DatafeedJob.java:424) ~[x-pack-ml-7.5.0.jar:7.5.0]
        at org.elasticsearch.xpack.ml.datafeed.DatafeedJob.runRealtime(DatafeedJob.java:196) ~[x-pack-ml-7.5.0.jar:7.5.0]
        at org.elasticsearch.xpack.ml.datafeed.DatafeedManager$Holder.executeRealTime(DatafeedManager.java:421) ~[x-pack-ml-7.5.0.jar:7.5.0]
        at org.elasticsearch.xpack.ml.datafeed.DatafeedManager$Holder.access$600(DatafeedManager.java:304) [x-pack-ml-7.5.0.jar:7.5.0]
        at org.elasticsearch.xpack.ml.datafeed.DatafeedManager$3.doRun(DatafeedManager.java:234) [x-pack-ml-7.5.0.jar:7.5.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.0.jar:7.5.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.0.jar:7.5.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2019-12-17T16:31:55,110][INFO ][o.e.x.m.d.DatafeedManager] [d1r2n14] [no_data] datafeed [datafeed-windows_anomalous_path_activity_ecs] for job [windows_anomalous_path_activity_ecs] has been stopped
[2019-12-17T16:31:55,334][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [d1r2n14] Closing job [windows_anomalous_path_activity_ecs], because [close job (api)]
[2019-12-17T16:31:55,335][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [d1r2n14] [windows_anomalous_path_activity_ecs] [autodetect/26826] [CCmdSkeleton.cc@45] Handled 0 records
[2019-12-17T16:31:55,335][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [d1r2n14] [windows_anomalous_path_activity_ecs] [autodetect/26826] [CAnomalyJob.cc@1499] Pruning all models
[2019-12-17T16:31:55,339][INFO ][o.e.x.m.p.AbstractNativeProcess] [d1r2n14] [windows_anomalous_path_activity_ecs] State output finished
[2019-12-17T16:31:55,361][INFO ][o.e.x.m.j.p.a.o.AutodetectResultProcessor] [d1r2n14] [windows_anomalous_path_activity_ecs] 0 buckets parsed from autodetect output
[2019-12-17T16:31:55,416][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [d1r2n14] [windows_anomalous_path_activity_ecs] job closed
2

Hi Steffen, it looks like the datafeed is not receiving any data:

Datafeed for [windows_anomalous_path_activity_ecs] has seen no data in [10] attempts, and never seen any data previously, so stopping...

This particular job filters the winlogbeat data with the following query:

"query": {
        "bool": {
          "filter": [
            {"term": {"event.action": "Process Create (rule: ProcessCreate)"}},
            {"term": {"agent.type": "winlogbeat"}}
          ]
        }
      }

Can you run this against your winlogbeat data? If this query does not return any data, then the datafeed will stop - so what you're experiencing is expected behavior.

3

@blaklaybul :slight_smile: missed out on that specific log event:

Datafeed for [windows_anomalous_path_activity_ecs] has seen no data in [10] attempts, and never seen any data previously, so stopping...

Though it's weird not just one process creation should yet be seen among our initial 12-14 agents shipping winlogbeat data in so far... wondering if the event.action is correct... or if it's because many EPs are logging in a different loca than english, so assume all ML jobs may need localization adaption...

danish for 'Process Create' seen in event.action: Procesoprettelse
.. bugger, thanks for pointing me in the right direction!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.