Machine Learning Real-time search doesn't work

serive · November 29, 2017, 3:10am

When I made single metric job with
Aggregation: count
Bucket span: 10m
and start datafeed "No end time (Real-time search)",
real-time search didn't start at all.

At that time, I stopped datafeeed, edit job to change frequency as 10m and start datafeed again.
Then the datafeed started correctly.

The default value of frequency was 300s.
Do I need to set frequency greater than bucket span?

Elasticsearch version is 6.0.0.
Here is the settings of my test job.

{
  "job_id": "test2_job_bucketspan10m",
  "job_type": "anomaly_detector",
  "job_version": "6.0.0",
  "description": "bucket span 10m",
  "create_time": 1511851800489,
  "finished_time": 1511851801975,
  "analysis_config": {
    "bucket_span": "10m",
    "summary_count_field_name": "doc_count",
    "detectors": [
      {
        "detector_description": "count",
        "function": "count",
        "detector_rules": [],
        "detector_index": 0
      }
    ],
    "influencers": []
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_ms"
  },
  "model_plot_config": {
    "enabled": true
  },
  "model_snapshot_retention_days": 1,
  "model_snapshot_id": "1511909470",
  "results_index_name": "shared",
  "data_counts": {
    "job_id": "test2_job_bucketspan10m",
    "processed_record_count": 874,
    "processed_field_count": 874,
    "input_bytes": 40088,
    "input_field_count": 874,
    "invalid_date_count": 0,
    "missing_field_count": 0,
    "out_of_order_timestamp_count": 0,
    "empty_bucket_count": 0,
    "sparse_bucket_count": 0,
    "bucket_count": 872,
    "earliest_record_timestamp": 1511395799640,
    "latest_record_timestamp": 1511923199983,
    "last_data_time": 1511923270691,
    "input_record_count": 874
  },
  "model_size_stats": {
    "job_id": "test2_job_bucketspan10m",
    "result_type": "model_size_stats",
    "model_bytes": 79064,
    "total_by_field_count": 3,
    "total_over_field_count": 0,
    "total_partition_field_count": 2,
    "bucket_allocation_failures_count": 0,
    "memory_status": "ok",
    "log_time": 1511910070000,
    "timestamp": 1511909400000
  },
  "datafeed_config": {
    "datafeed_id": "datafeed-test2_job_bucketspan10m",
    "job_id": "test2_job_bucketspan10m",
    "query_delay": "70587ms",
    "frequency": "600s",
    "indices": [
      "test2-*"
    ],
    "types": [
      "doc"
    ],
    "query": {
      "match_all": {
        "boost": 1
      }
    },
    "aggregations": {
      "buckets": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": 600000,
          "offset": 0,
          "order": {
            "_key": "asc"
          },
          "keyed": false,
          "min_doc_count": 0
        },
        "aggregations": {
          "@timestamp": {
            "max": {
              "field": "@timestamp"
            }
          }
        }
      }
    },
    "scroll_size": 1000,
    "chunking_config": {
      "mode": "manual",
      "time_span": "600000000ms"
    },
    "state": "started",
    "node": {
      "id": "WMwq9r88Rq2P3Etj1pmfzQ",
      "name": "WMwq9r8",
      "ephemeral_id": "iYzswhp9SymdvkDe9LOl-g",
      "transport_address": "127.0.0.1:9300",
      "attributes": {
        "ml.max_open_jobs": "10",
        "ml.enabled": "true"
      }
    }
  },
  "state": "opened",
  "node": {
    "id": "WMwq9r88Rq2P3Etj1pmfzQ",
    "name": "WMwq9r8",
    "ephemeral_id": "iYzswhp9SymdvkDe9LOl-g",
    "transport_address": "127.0.0.1:9300",
    "attributes": {
      "ml.max_open_jobs": "10",
      "ml.enabled": "true"
    }
  },
  "open_time": "71962s"
}

dkyle · November 29, 2017, 10:21am

Hi can you tell me exactly what the error message was please, even better if you can share your elasticsearch log file.

The frequency can be less than the bucket span, I would usually recommend that it is <= to the bucket span.

serive · November 30, 2017, 2:27am

This is how I created new Simple metric job.

Aggregation: Count
Bucket span: 10m
Continue job in realtime

The test data is already made and indexed since now-7d to now+7d.

First datafeed from now-7d to now works fine.
But the realtime datafeed does not work.

There is no log message after starting datafeed.

elasticsearch.log

[2017-11-30T01:47:50,261][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Opening job [test1]
[2017-11-30T01:47:50,263][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] [test1] Loading model snapshot [N/A], job latest_record_timestamp [N/A]
[2017-11-30T01:47:50,359][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Successfully set job state to [opened] for job [test1]
[2017-11-30T01:47:50,462][INFO ][o.e.x.m.a.PutDatafeedAction$TransportAction] [WMwq9r8] Created datafeed [datafeed-test1]
[2017-11-30T01:47:50,591][INFO ][o.e.x.m.d.DatafeedManager] Starting datafeed [datafeed-test1] for job [test1] in [2017-11-23T00:01:59.240Z, 2017-11-30T01:44:59.541Z)
[2017-11-30T01:47:51,913][INFO ][o.e.x.m.d.DatafeedJob    ] [test1] Lookback has finished
[2017-11-30T01:47:51,913][INFO ][o.e.x.m.d.DatafeedManager] [no_realtime] attempt to stop datafeed [datafeed-test1] for job [test1]
[2017-11-30T01:47:51,913][INFO ][o.e.x.m.d.DatafeedManager] [no_realtime] try lock [20s] to stop datafeed [datafeed-test1] for job [test1]...
[2017-11-30T01:47:51,914][INFO ][o.e.x.m.d.DatafeedManager] [no_realtime] stopping datafeed [datafeed-test1] for job [test1], acquired [true]...
[2017-11-30T01:47:51,914][INFO ][o.e.x.m.d.DatafeedManager] [no_realtime] datafeed [datafeed-test1] for job [test1] has been stopped
[2017-11-30T01:47:51,942][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Closing job [test1], because [close job (api)]
[2017-11-30T01:47:51,942][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [test1] [autodetect/3254] [CCmdSkeleton.cc@70] Handled 1017 records
[2017-11-30T01:47:51,943][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [test1] [autodetect/3254] [CAnomalyDetector.cc@1704] Pruning all models
[2017-11-30T01:47:51,950][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcess] [test1] State output finished
[2017-11-30T01:47:51,978][INFO ][o.e.x.m.j.p.a.o.AutoDetectResultProcessor] [test1] 1016 buckets parsed from autodetect output
[2017-11-30T01:47:52,132][INFO ][o.e.x.m.j.p.a.AutodetectCommunicator] [test1] job closed
[2017-11-30T01:47:56,104][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Opening job [test1]
[2017-11-30T01:47:56,108][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] [test1] Loading model snapshot [1512006471] with latest_record_timestamp [2017-11-30T01:39:59.000Z], job latest_record_timestamp [2017-11-30T01:39:59.920Z]
[2017-11-30T01:47:56,109][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcessFactory] Restoring quantiles for job 'test1'
[2017-11-30T01:47:56,187][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [WMwq9r8] Successfully set job state to [opened] for job [test1]
[2017-11-30T01:47:56,199][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [test1] [autodetect/3260] [CAnomalyDetector.cc@976] Processing is already complete to time 1512005400
[2017-11-30T01:47:56,306][INFO ][o.e.x.m.d.DatafeedManager] Starting datafeed [datafeed-test1] for job [test1] in [1970-01-01T00:00:00.000Z, forever)

dkyle · December 1, 2017, 4:56pm

HI @serive

I haven't been able to recreate your problem. The steps I took are:

Index data for the time period now -7d to now +7d
Create a single metric job and set the end time to now in the date range picker
Click Create Job and wait for the analysis to finish
Select Continue job in realtime and click apply

And the job continued in real time successfully.

In the second picture you posted the fact that we see the light blue area representing the model bounds indicates the datafeed is working but the missing dark blue line suggests the data is missing

dmitri · December 1, 2017, 5:21pm

Hi,

Do your data have an explicit timezone in the time field? If not, could you please retry your test but this time add explicit timezone information?

dmitri · December 1, 2017, 7:57pm

Hi,

Ignore my comment above.

I managed to reproduce the problem you observe. I will raise an issue internally so the issue is fixed in future release.

For now please set the frequency to be equal to the aggregation histogram in the datafeed, i.e 10 minutes in your case.

Thank you for the valuable feedback, it makes a huge difference into making our products better.

serive · December 4, 2017, 1:17am

Thanks!

system · January 1, 2018, 1:21am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Advanced job can not be 'realtime search' with xpack5.4 machine learning Elasticsearch elastic-stack-machine-learning	10	1339	July 21, 2017
Datafeed frequency must be a multiple of the aggregation interval Elasticsearch elastic-stack-machine-learning	3	896	October 30, 2018
[6.1.1] Machine Learning Job: "Rolling" Datafeed Period Elasticsearch elastic-stack-machine-learning	2	616	March 10, 2018
ML job not updating in real time Elasticsearch elastic-stack-machine-learning	4	859	October 29, 2018
ML datafeed with bursty data Elasticsearch elastic-stack-machine-learning	4	600	October 29, 2018

Machine Learning Real-time search doesn't work

Related topics