# Advanced job can not be 'realtime search' with xpack5.4 machine learning

**URL:** <https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [June 20, 2017, 4:16am UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023 "2017-06-20T04:16:10Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![19e5057b4cab9cf379c9](https://avatars.discourse-cdn.com/v4/letter/1/2bfe46/32.png) [@19e5057b4cab9cf379c9](https://discuss.elastic.co/u/19e5057b4cab9cf379c9)\
**Post date:** [June 20, 2017, 4:16am UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/1 "2017-06-20T04:16:10Z")

</div>

hey,gays

when i created advanced job and chose ‘no end time(realtime search)’ . if data load finish, the process will

sleep. after about 5min data keep push to elasticsearch ,but the process is not work.

help me~ thx.

![](https://us1.discourse-cdn.com/elastic/original/3X/9/7/97712c94cb2dc17a42c4e7aa3f7641d10667c564.png)

 ![](https://us1.discourse-cdn.com/elastic/original/3X/5/6/5630315cba4c5c3d6eea32f380c4b569afd5d62e.png)

 ![](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a3c02c1a229e836a5be2ea5de7d1205767e5a0a8.png)

---

<div class="post-metadata">

**Author:** ![dmitri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmitri/32/17997_2.png) [@dmitri](https://discuss.elastic.co/u/dmitri)\
**Post date:** [June 20, 2017, 10:35am UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/2 "2017-06-20T10:35:40Z")

</div>

Hello,

Could you please paste the job/datafeed configuration? It is a necessary first step to figure what is going on.

Thank you,  
Dimitris

---

<div class="post-metadata">

**Author:** ![19e5057b4cab9cf379c9](https://avatars.discourse-cdn.com/v4/letter/1/2bfe46/32.png) [@19e5057b4cab9cf379c9](https://discuss.elastic.co/u/19e5057b4cab9cf379c9)\
**Post date:** [June 20, 2017, 11:57am UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/3 "2017-06-20T11:57:18Z")

</div>

hi，dmitri.  
 ![](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4cd123b118a144d58b8e5478aaca23a4da7b084d.png)  
 ![](https://us1.discourse-cdn.com/elastic/original/3X/8/6/86ef6c8f978c4010f7735ebd526ffee16067566d.png)

 ![](https://us1.discourse-cdn.com/elastic/original/3X/9/d/9d463c4745c3174510ecf1e4f12a6e5055fed5b2.png)

datafeed configuration is default. i try to set the ’Query delay‘,'Frequency' larger. like 300s , 900s ,this job can be kept updated during each cycle.but default configuration can not work, also remind me 'Datafeed has been retrieving no data for a while'. elasticsearch index realtime receive data, i check over the index has new record.

thank you  
ray

---

<div class="post-metadata">

**Author:** ![dmitri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmitri/32/17997_2.png) [@dmitri](https://discuss.elastic.co/u/dmitri)\
**Post date:** [June 20, 2017, 12:03pm UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/4 "2017-06-20T12:03:16Z")

</div>

Hi Ray,

Thank you for this. In order to get the full picture, could you please paste the JSON from the JSON tab of the job?

Thank you,  
Dimitris

---

<div class="post-metadata">

**Author:** ![19e5057b4cab9cf379c9](https://avatars.discourse-cdn.com/v4/letter/1/2bfe46/32.png) [@19e5057b4cab9cf379c9](https://discuss.elastic.co/u/19e5057b4cab9cf379c9)\
**Post date:** [June 20, 2017, 1:41pm UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/5 "2017-06-20T13:41:32Z")

</div>

hi dmitri.

```auto
{
    "job_id": "test4",
    "job_type": "anomaly_detector",
    "description": "test4",
    "create_time": 1496515790240,
    "analysis_config": {
        "bucket_span": "5m",
        "detectors": [
            {
                "detector_description": "high_sum(session_down) (test4)",
                "function": "high_sum",
                "field_name": "session_down",
                "detector_rules": []
            },
            {
                "detector_description": "high_sum(session_up) (test4)",
                "function": "high_sum",
                "field_name": "session_up",
                "detector_rules": []
            }
        ],
        "influencers": []
    },
    "data_description": {
        "time_field": "session_start_time",
        "time_format": "epoch_ms"
    },
    "model_snapshot_retention_days": 1,
    "model_snapshot_id": "1497965270",
    "results_index_name": "shared",
    "data_counts": {
        "job_id": "test4",
        "processed_record_count": 48131071,
        "processed_field_count": 96262142,
        "input_bytes": 3446702239,
        "input_field_count": 96262142,
        "invalid_date_count": 0,
        "missing_field_count": 0,
        "out_of_order_timestamp_count": 0,
        "empty_bucket_count": 0,
        "sparse_bucket_count": 305,
        "bucket_count": 71350,
        "earliest_record_timestamp": 1494049387000,
        "latest_record_timestamp": 1497964540000,
        "last_data_time": 1497965275678,
        "latest_empty_bucket_timestamp": 1497168300000,
        "latest_sparse_bucket_timestamp": 1497963600000,
        "input_record_count": 48131071
    },
    "model_size_stats": {
        "job_id": "test4",
        "result_type": "model_size_stats",
        "model_bytes": 85744,
        "total_by_field_count": 4,
        "total_over_field_count": 0,
        "total_partition_field_count": 3,
        "bucket_allocation_failures_count": 0,
        "memory_status": "ok",
        "log_time": 1497950997000,
        "timestamp": 1497651000000
    },
    "datafeed_config": {
        "datafeed_id": "datafeed-test4",
        "job_id": "test4",
        "query_delay": "60s",
        "frequency": "150s",
        "indexes": [
            "session*"
        ],
        "types": [
            "sessionmin_log"
        ],
        "query": {
            "match_all": {
                "boost": 1
            }
        },
        "scroll_size": 1000,
        "chunking_config": {
            "mode": "auto"
        },
        "state": "started"
    },
    "state": "opened",
    "node": {
        "id": "oBFK-X6pRd-TYm-WNa_BrA",
        "name": "xjtu-bigdata01",
        "ephemeral_id": "L5i9lVVnTEKnLE3KeHg73w",
        "transport_address": "172.16.0.11:9300",
        "attributes": {
            "ml.enabled": "true"
        }
    },
    "open_time": "1449747s"
}

```

I searched for the last record in the index， "session\_start\_time"=\>"2017-06-20T21:27:37.000+08:00" , "total"=\>48170388.

thank you  
ray

---

<div class="post-metadata">

**Author:** ![dmitri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmitri/32/17997_2.png) [@dmitri](https://discuss.elastic.co/u/dmitri)\
**Post date:** [June 20, 2017, 2:16pm UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/6 "2017-06-20T14:16:18Z")

</div>

Hi Ray,

Thank you for that.

As a next step, we need to take a look at the logs. However, ML datafeeds do not log much on the info level. Thus, we need to enable trace logging for datafeeds. You can achieve that by editing the logging configuration file which can usually be found at `/path-to-elasticsearch/config/log4j2.properties`.

In there, could you please add the following 2 lines:

```
logger.datafeed.name = org.elasticsearch.xpack.ml.datafeed
logger.datafeed.level = trace

```

You will need to restart your cluster in order for this change to apply.

Once that is done, could you please repeat your test and paste the output from the `elasticsearch.log` file?

Thank you,  
Dimitris

---

<div class="post-metadata">

**Author:** ![dmitri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmitri/32/17997_2.png) [@dmitri](https://discuss.elastic.co/u/dmitri)\
**Post date:** [June 21, 2017, 3:18pm UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/7 "2017-06-21T15:18:37Z")

</div>

Another thing that would be very useful, is to double check that the time field you use in your data has the correct timezone information. Note that if you index a date field without timezone information, elasticsearch assumes it is UTC. Thus, it is possible that the datafeed skips over that data, or that it will be running a few hours behind the data's timestamp.

You could check that by running a search against your data where you add:

```auto
{
  ...
  "docvalue_fields": ["{your_time_field}"],
  ...

```

If you could post an example document it would be great to further understand what the issue is here.

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [June 21, 2017, 8:47pm UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/8 "2017-06-21T20:47:19Z")

</div>

Ray,

I see from your configuration:

> [@](#):
>
> "data\_description": {  
> "time\_field": "session\_start\_time",  
> "time\_format": "epoch\_ms"  
> },

that you are using `session_start_time` as your time field for the job. I'm not familiar with your data, but I can imagine a situation in which when the actual document is inserted into Elasticsearch, this `session_start_time` field could reference a time a few minutes in the past (especially if the "sessions" that your tracking last for more than a few minutes). If that's the case, then settings currently being used for your ML job's datafeed may NEVER see your data. Here's an explanation as to why:

Let's say at 1:00:30 PM local time, a document is ingested into Elasticsearch, and the field `session_start_time` has a value that's equivalent to 12:49:00 PM. And let's pretend there's another field `session_end_time` that has a value of 1:00:00 PM (thus the session's duration was 11 minutes). (I'm presuming that the document is written and ingested once the session is over). By this logic, it also took 30 seconds from the document's creation on the device (1:00:00 PM) before it was ingested by Elasticsearch (at 1:00:30 PM). This is the ingest pipeline delay.

Let's now imagine a separate situation where at 1:00:00 PM, I query Elasticsearch for documents where `session_end_time` is between 12:55:00 PM and 1:00:00 PM) - a 5 minute "bucket\_span". If I did this, would the document described above even exist in Elasticsearch? The answer is no, as it will only be indexed 30 seconds from now. But if I delayed my looking for that document by 60s (at 1:01:00 PM), would I find it? Well, the document would indeed exist in Elasticsearch by that time (it has existed for 30 seconds at this point), but the query itself won't find it - because the query is asking for matches on values of `session_start_time` being between 12:55:00 PM and 1:00:00 PM. Our document has a `session_start_time` of 12:49:00 PM. So, it's already/instantly outside of the range of time we're looking for!

So, if this is the possible scenario, the way around this is to increase the `query_delay` parameter. The default value of 60s is not enough. In the scenario I describe above, where sessions can run on the order of 10-15 minutes, I may need to bump the `query_delay` parameter up to perhaps 20m. Then, at 1:05:00 PM, the query will run asking for matches of `session_start_time` being between 12:45:00 PM and 12:50:00 PM - and it'll find my document where `session_start_time` is 12:49:00 PM

Hope that helps

---

<div class="post-metadata">

**Author:** ![19e5057b4cab9cf379c9](https://avatars.discourse-cdn.com/v4/letter/1/2bfe46/32.png) [@19e5057b4cab9cf379c9](https://discuss.elastic.co/u/19e5057b4cab9cf379c9)\
**Post date:** [June 23, 2017, 7:05am UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/9 "2017-06-23T07:05:08Z")

</div>

hi,dmitri  
thank you for help，I guess problem is time filed delayed.

i will go on check this. ^ ^

---

<div class="post-metadata">

**Author:** ![19e5057b4cab9cf379c9](https://avatars.discourse-cdn.com/v4/letter/1/2bfe46/32.png) [@19e5057b4cab9cf379c9](https://discuss.elastic.co/u/19e5057b4cab9cf379c9)\
**Post date:** [June 23, 2017, 7:13am UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/10 "2017-06-23T07:13:01Z")

</div>

hi，richcollier  
thankyou for help, nice explanation, let me know the job how to work. 66666666666!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 21, 2017, 7:13am UTC](https://discuss.elastic.co/t/advanced-job-can-not-be-realtime-search-with-xpack5-4-machine-learning/90023/11 "2017-07-21T07:13:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.