Machine Learning Real-time search doesn't work

Elastic Stack Elasticsearch
1

When I made single metric job with
Aggregation: count
Bucket span: 10m
and start datafeed "No end time (Real-time search)",
real-time search didn't start at all.

At that time, I stopped datafeeed, edit job to change frequency as 10m and start datafeed again.
Then the datafeed started correctly.

The default value of frequency was 300s.
Do I need to set frequency greater than bucket span?

Elasticsearch version is 6.0.0.
Here is the settings of my test job.

{
  "job_id": "test2_job_bucketspan10m",
  "job_type": "anomaly_detector",
  "job_version": "6.0.0",
  "description": "bucket span 10m",
  "create_time": 1511851800489,
  "finished_time": 1511851801975,
  "analysis_config": {
    "bucket_span": "10m",
    "summary_count_field_name": "doc_count",
    "detectors": [
      {
        "detector_description": "count",
        "function": "count",
        "detector_rules": [],
        "detector_index": 0
      }
    ],
    "influencers": []
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_ms"
  },
  "model_plot_config": {
    "enabled": true
  },
  "model_snapshot_retention_days": 1,
  "model_snapshot_id": "1511909470",
  "results_index_name": "shared",
  "data_counts": {
    "job_id": "test2_job_bucketspan10m",
    "processed_record_count": 874,
    "processed_field_count": 874,
    "input_bytes": 40088,
    "input_field_count": 874,
    "invalid_date_count": 0,
    "missing_field_count": 0,
    "out_of_order_timestamp_count": 0,
    "empty_bucket_count": 0,
    "sparse_bucket_count": 0,
    "bucket_count": 872,
    "earliest_record_timestamp": 1511395799640,
    "latest_record_timestamp": 1511923199983,
    "last_data_time": 1511923270691,
    "input_record_count": 874
  },
  "model_size_stats": {
    "job_id": "test2_job_bucketspan10m",
    "result_type": "model_size_stats",
    "model_bytes": 79064,
    "total_by_field_count": 3,
    "total_over_field_count": 0,
    "total_partition_field_count": 2,
    "bucket_allocation_failures_count": 0,
    "memory_status": "ok",
    "log_time": 1511910070000,
    "timestamp": 1511909400000
  },
  "datafeed_config": {
    "datafeed_id": "datafeed-test2_job_bucketspan10m",
    "job_id": "test2_job_bucketspan10m",
    "query_delay": "70587ms",
    "frequency": "600s",
    "indices": [
      "test2-*"
    ],
    "types": [
      "doc"
    ],
    "query": {
      "match_all": {
        "boost": 1
      }
    },
    "aggregations": {
      "buckets": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": 600000,
          "offset": 0,
          "order": {
            "_key": "asc"
          },
          "keyed": false,
          "min_doc_count": 0
        },
        "aggregations": {
          "@timestamp": {
            "max": {
              "field": "@timestamp"
            }
          }
        }
      }
    },
    "scroll_size": 1000,
    "chunking_config": {
      "mode": "manual",
      "time_span": "600000000ms"
    },
    "state": "started",
    "node": {
      "id": "WMwq9r88Rq2P3Etj1pmfzQ",
      "name": "WMwq9r8",
      "ephemeral_id": "iYzswhp9SymdvkDe9LOl-g",
      "transport_address": "127.0.0.1:9300",
      "attributes": {
        "ml.max_open_jobs": "10",
        "ml.enabled": "true"
      }
    }
  },
  "state": "opened",
  "node": {
    "id": "WMwq9r88Rq2P3Etj1pmfzQ",
    "name": "WMwq9r8",
    "ephemeral_id": "iYzswhp9SymdvkDe9LOl-g",
    "transport_address": "127.0.0.1:9300",
    "attributes": {
      "ml.max_open_jobs": "10",
      "ml.enabled": "true"
    }
  },
  "open_time": "71962s"
}
2

Hi can you tell me exactly what the error message was please, even better if you can share your elasticsearch log file.

The frequency can be less than the bucket span, I would usually recommend that it is <= to the bucket span.

3

This is how I created new Simple metric job.

Aggregation: Count
Bucket span: 10m
Continue job in realtime

2017-11-30_104824
2017-11-30_104824.png1406×898 76.8 KB

The test data is already made and indexed since now-7d to now+7d.

First datafeed from now-7d to now works fine.
But the realtime datafeed does not work.

2017-11-30_112325
2017-11-30_112325.png1406×898 57.5 KB

There is no log message after starting datafeed.

elasticsearch.log

[2017-11-30T01:47:50,261][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Opening job [test1]
[2017-11-30T01:47:50,263][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] [test1] Loading model snapshot [N/A], job latest_record_timestamp [N/A]
[2017-11-30T01:47:50,359][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Successfully set job state to [opened] for job [test1]
[2017-11-30T01:47:50,462][INFO ][o.e.x.m.a.PutDatafeedAction$TransportAction] [WMwq9r8] Created datafeed [datafeed-test1]
[2017-11-30T01:47:50,591][INFO ][o.e.x.m.d.DatafeedManager] Starting datafeed [datafeed-test1] for job [test1] in [2017-11-23T00:01:59.240Z, 2017-11-30T01:44:59.541Z)
[2017-11-30T01:47:51,913][INFO ][o.e.x.m.d.DatafeedJob    ] [test1] Lookback has finished
[2017-11-30T01:47:51,913][INFO ][o.e.x.m.d.DatafeedManager] [no_realtime] attempt to stop datafeed [datafeed-test1] for job [test1]
[2017-11-30T01:47:51,913][INFO ][o.e.x.m.d.DatafeedManager] [no_realtime] try lock [20s] to stop datafeed [datafeed-test1] for job [test1]...
[2017-11-30T01:47:51,914][INFO ][o.e.x.m.d.DatafeedManager] [no_realtime] stopping datafeed [datafeed-test1] for job [test1], acquired [true]...
[2017-11-30T01:47:51,914][INFO ][o.e.x.m.d.DatafeedManager] [no_realtime] datafeed [datafeed-test1] for job [test1] has been stopped
[2017-11-30T01:47:51,942][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Closing job [test1], because [close job (api)]
[2017-11-30T01:47:51,942][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [test1] [autodetect/3254] [CCmdSkeleton.cc@70] Handled 1017 records
[2017-11-30T01:47:51,943][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [test1] [autodetect/3254] [CAnomalyDetector.cc@1704] Pruning all models
[2017-11-30T01:47:51,950][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcess] [test1] State output finished
[2017-11-30T01:47:51,978][INFO ][o.e.x.m.j.p.a.o.AutoDetectResultProcessor] [test1] 1016 buckets parsed from autodetect output
[2017-11-30T01:47:52,132][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [test1] job closed
[2017-11-30T01:47:56,104][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Opening job [test1]
[2017-11-30T01:47:56,108][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] [test1] Loading model snapshot [1512006471] with latest_record_timestamp [2017-11-30T01:39:59.000Z], job latest_record_timestamp [2017-11-30T01:39:59.920Z]
[2017-11-30T01:47:56,109][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcessFactory] Restoring quantiles for job 'test1'
[2017-11-30T01:47:56,187][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Successfully set job state to [opened] for job [test1]
[2017-11-30T01:47:56,199][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [test1] [autodetect/3260] [CAnomalyDetector.cc@976] Processing is already complete to time 1512005400
[2017-11-30T01:47:56,306][INFO ][o.e.x.m.d.DatafeedManager] Starting datafeed [datafeed-test1] for job [test1] in [1970-01-01T00:00:00.000Z, forever)
4

HI @serive

I haven't been able to recreate your problem. The steps I took are:

  1. Index data for the time period now -7d to now +7d
  2. Create a single metric job and set the end time to now in the date range picker
  3. Click Create Job and wait for the analysis to finish
  4. Select Continue job in realtime and click apply

And the job continued in real time successfully.

In the second picture you posted the fact that we see the light blue area representing the model bounds indicates the datafeed is working but the missing dark blue line suggests the data is missing

5

Hi,

Do your data have an explicit timezone in the time field? If not, could you please retry your test but this time add explicit timezone information?

6

Hi,

Ignore my comment above.

I managed to reproduce the problem you observe. I will raise an issue internally so the issue is fixed in future release.

For now please set the frequency to be equal to the aggregation histogram in the datafeed, i.e 10 minutes in your case.

Thank you for the valuable feedback, it makes a huge difference into making our products better.

7

Thanks!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.