Incorrect Timestamp in Machine Learning Job Result

Elastic Stack Elasticsearch
1

Hi,

I have a few documents that have a timestamp of 01:46:00:038. But, after running a ML job, the results show these documents with a different time 07:16:00:038. I guess this is due to time difference of 5:30 but I do not know why this is happening. Can you please provide a solution to fix this issue?

Thanks!

2

Kibana will show events in your local timezone unless you change that option:

https://www.elastic.co/guide/en/kibana/current/advanced-options.html

3

Hi,

I changed the time zone to GMT. Now, @timestamp matches the timestamp in the log file. However, in the machine learning module, the timestamp is still showing in GMT+5:30 (my local time).

Why isnt the machine learning job taking the value in @timestamp?

4

Hi Niket,

Just to clarify - did you change the time zone to GMT for the 'dateFormat:tz' under the Advanced Settings tab of the Kibana Management page?

Unfortunately the visuals in the ML plugin will only currently display times in the local timezone of your browser (with dateFormat:tz set to the default of 'Browser'). This is something we will look to address in a subsequent release.

Thanks
Pete

1 Like
5

Hi Peter,

Yes, I did change the time zone to GMT for the ‘dateFormat:tz’. When I want to send alerts for anomalies detected, the timestamp for those anomalies is incorrect due to the ML result. Can you help me solve this? If not then I'll wait for the next release.

Thanks!

6

Is it possible that the data you are analyzing contains timestamps in text format in the India timezone but without the timezone in the textual representation? In this case you'll end up with a _source document that shows your original textual representation, which you'll interpret as India time. But Elasticsearch will have converted it to epoch milliseconds to store in the doc values on the assumption it's UTC. Then ML will analyze it and create results based on the epoch milliseconds stored in the doc values.

This thread discusses the way Elasticsearch converts textual timestamps assuming UTC unless the text contains a timezone: Storing Dates in Elastic Best Practices

If what I'm suggesting isn't clear then it might be easier to discuss a specific example. If you paste the _source of one of the documents you're analyzing then I can talk specifically about it.

7

Here is a sample _source document:

"_source": {
          "job_id": "test_job",
          "result_type": "record",
          "probability": 0.00684734,
          "record_score": 35.3922,
          "initial_record_score": 35.3922,
          "bucket_span": 300,
          "detector_index": 1,
          "is_interim": false,
          "timestamp": 1503741600000,
          "by_field_name": "mlcategory",
          "by_field_value": "19",
          "function": "non_zero_count",
          "function_description": "count",
          "typical": [
            1.03523
          ],
          "actual": [
            6
          ],
          "influencers": [
            {
              "influencer_field_name": "sampler.keyword",
              "influencer_field_values": [
                "101"
              ]
            }
          ],
          "sampler.keyword": [
            "101"
          ],
          "mlcategory": [
            "19"
          ]
        }
8

Ah, OK. So your timestamps are already in milliseconds since the epoch in the source documents. So timezone conversions on indexing cannot be the problem. My theory was wrong then.

9

OK, so seeing as timezone conversions on indexing aren't the root of the problem, hopefully your issue will be addressed when we fix the ML pages to observe the Kibana ‘dateFormat:tz’ configuration option when set to something other than 'Browser'. Till then the times in the ML pages will be offset to GMT according to the local timezone of your browser. I'll provide an update when that is fixed.

10

OK. I'll write a small note regarding the timestamp issue while alerting for now. I hope this issue gets resolved soon.

Thanks Peter!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.