Changing UTC to local time zone in a machine learning watcher

Elastic Stack Elasticsearch
1

Im trying to get the time of the alert to local time, but when I add the transform to the watcher I get the mail and the date, but the results are gone, it seems that the transform alters the entire result, what can I do to avoid this?

"transform": {
        "script": {
          "source": """
              DateTimeFormatter dtf = DateTimeFormatter.ofPattern("yyyy/MM/dd HH:mm:ss");
              return ['date': Instant.ofEpochMilli(OffsetDateTime.parse(ctx.payload.aggregations.bucket_results.top_bucket_hits.hits.hits.0.fields.timestamp_iso8601.0).toInstant().toEpochMilli()).atZone(ZoneId.of("America/Santiago")).format(dtf)]""",
          "lang": "painless"
        }
      }

image

the transform is inside the action section of the watcher

2

At a glance it seems to me that your script only returns one field ('date').
That's why the other fields are lost.

              return ['date': Instant.ofEpochMilli(OffsetDateTime.parse(ctx.payload.aggregations.bucket_results.top_bucket_hits.hits.hits.0.fields.timestamp_iso8601.0).toInstant().toEpochMilli()).atZone(ZoneId.of("America/Santiago")).format(dtf)]""",
1 Like
3

Przemek is 100% correct. If you use a transform, you must return everything that you want to report on in the email or slack or log message...

For example, see: https://gist.github.com/richcollier/1c2b8161286bdca6c553859f28d3d66d

Notice in the transform:

        "transform": {
          "script": """
          return ctx.payload.hits.hits.stream()
          .map(p -> [
            'airline':p.fields.split.0,
            'score':p.fields.score.0,
            'actual':p.fields.actual.0,
            'typical':p.fields.typical.0,
            'timestamp':p.fields.timestamp_iso8601.0,
            'start':p.fields.start.0,
            'end':p.fields.end.0
            ])
            .collect(Collectors.toList());
"""
        },

The context (the ctx object) is referring to the output of the query to the .ml-anomalies-* index. This is why the payload has a nested hits array.

But, after the transform, a "new" context is made (which will be the output of the transform). Therefore, in the logging section:

        "logging": {
          "text": """
Anomalies:
==========
{{#ctx.payload._value}}
time={{timestamp}} 
airline={{airline}} 
score={{score}} (out of 100) 
responsetime={{actual}}ms (typical={{typical}}ms)
link= http://localhost:5601/app/ml#/timeseriesexplorer/?_g=(ml:(jobIds:!({{ctx.metadata.job_id}})),refreshInterval:(display:Off,pause:!f,value:0),time:(from:'{{start}}',mode:absolute,to:'{{end}}'))&_a=(filters:!(),mlSelectInterval:(interval:(display:Auto,val:auto)),mlSelectSeverity:(threshold:(display:warning,val:0)),mlTimeSeriesExplorer:(detectorIndex:0,entities:(airline:{{airline}})),query:(query_string:(analyze_wildcard:!t,query:'*')))
{{/ctx.payload._value}}
"""
        }
      }

Notice that the ctx object now (after the transform) has more "simple" variable references (such as timestamp, score, airline, etc.) - because, that's the names of the variables created in the fancy transform script.

When run in DevTools, the output seen looks like:

      "actions" : [
        {
          "id" : "log",
          "type" : "logging",
          "status" : "success",
          "transform" : {
            "type" : "script",
            "status" : "success",
            "payload" : {
              "_value" : [
                {
                  "score" : 99,
                  "actual" : 282,
                  "typical" : 100,
                  "start" : "2017-02-09T13:45:00.000Z",
                  "end" : "2017-02-09T18:45:00.000Z",
                  "airline" : "AAL",
                  "timestamp" : "2017-02-09T16:15:00.000Z"
                },
                {
                  "score" : 92,
                  "actual" : 242,
                  "typical" : 100,
                  "start" : "2017-02-09T13:30:00.000Z",
                  "end" : "2017-02-09T18:30:00.000Z",
                  "airline" : "AAL",
                  "timestamp" : "2017-02-09T16:00:00.000Z"
                }
              ]
            }
          },
          "logging" : {
            "logged_text" : """
Anomalies:
==========
time=2017-02-09T16:15:00.000Z 
airline=AAL 
score=99 (out of 100) 
responsetime=282ms (typical=100ms)
link= http://localhost:5601/app/ml#/timeseriesexplorer/?_g=(ml:(jobIds:!(farequote_demo)),refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2017-02-09T13:45:00.000Z',mode:absolute,to:'2017-02-09T18:45:00.000Z'))&_a=(filters:!(),mlSelectInterval:(interval:(display:Auto,val:auto)),mlSelectSeverity:(threshold:(display:warning,val:0)),mlTimeSeriesExplorer:(detectorIndex:0,entities:(airline:AAL)),query:(query_string:(analyze_wildcard:!t,query:'*')))
time=2017-02-09T16:00:00.000Z 
airline=AAL 
score=92 (out of 100) 
responsetime=242ms (typical=100ms)
link= http://localhost:5601/app/ml#/timeseriesexplorer/?_g=(ml:(jobIds:!(farequote_demo)),refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2017-02-09T13:30:00.000Z',mode:absolute,to:'2017-02-09T18:30:00.000Z'))&_a=(filters:!(),mlSelectInterval:(interval:(display:Auto,val:auto)),mlSelectSeverity:(threshold:(display:warning,val:0)),mlTimeSeriesExplorer:(detectorIndex:0,entities:(airline:AAL)),query:(query_string:(analyze_wildcard:!t,query:'*')))
"""
          }
        }
      ]

Notice the ctx.payload._value object is a nice and tidy array of fields. The logging part of this particular watch then uses a mustache loop:

{{#ctx.payload._value}}
...stuff inside the loop...
{{/ctx.payload._value}}

To iterate through the ctx.payload._value array.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.