Iterate through influencer_field_values when they contain more than one value, that exists inside ctx.payload.hits.hits.stream()

Elastic Stack Kibana
, , ,
1

Consider the following output:

Anomaly on Sep 16 @ 11:45 PM:
Partner: p1             Brand: b1
Score=94.71           Actual=15    Typical=744.49

How do I go about adding more partners or brands, instead of getting this:

Anomaly on Sep 16 @ 11:45 PM:
Partner: {0=p1, 1=p2}            Brand: {0=b1, 1=b2}
Score=94.71           Actual=15    Typical=744.49

And End up with this:

==  Anomaly on Sep 16 @ 11:45 PM:  ==
Partner: p1, p2       Brand: b1, b2
Score=94.71           Actual=15    Typical=744.49

I can get the following:

partner=
p._source.influencers[0]['influencer_field_values'].0

or

brand=
p._source.influencers[1]['influencer_field_values'].0

if I actually specify it, but not all have more than 1, so how do I loop:

p._source.influencers[..]['influencer_field_values']

to display either:
[..]['influencer_field_values'].0
[..]['influencer_field_values'].1
[..]['influencer_field_values'].2
?

( where .. is either [0] or [1] depending on if I am trying to get partner or brand. The end (.0 or .1 or .2 ) is the part Im trying to figure out how to iterate.

Here is what I have to transform the output I have now:

  "transform": {
    "script": {
      "source": "def df = new DecimalFormat('##.##'); return [ 'anomaly_details':ctx.payload.hits.hits.stream().map(p -> ['start_time':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).minus(Duration.ofMinutes(20)).format(DateTimeFormatter.ofPattern('MMM d @ h:mm a')),'end_time':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).plus(Duration.ofMinutes(20)).format(DateTimeFormatter.ofPattern('MMM d @ h:mm a')),'bucket_time':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).format(DateTimeFormatter.ofPattern('MMM d @ h:mm a')),'job_id':p._source.job_id,'score':df.format(p._source.record_score),'actual':df.format(p._source.actual.0),'typical':df.format(p._source.typical.0),'partner':p._source.influencers[0]['influencer_field_values'].0,'brand':p._source.influencers[1]['influencer_field_values'].0,'start_time_iso_instant':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).minus(Duration.ofMinutes(20)).format(DateTimeFormatter.ISO_INSTANT),'end_time_iso_instant':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).plus(Duration.ofMinutes(20)).format(DateTimeFormatter.ISO_INSTANT) ]).collect(Collectors.toList())]",
      "lang": "painless"
    }
2

See example here: https://gist.github.com/richcollier/1c2b8161286bdca6c553859f28d3d66d

    "actions": {
      "log": {
        "transform": {
          "script": """
          return ctx.payload.hits.hits.stream()
          .map(p -> [
            'airline':p.fields.split.0,
            'score':p.fields.score.0,
            'actual':p.fields.actual.0,
            'typical':p.fields.typical.0,
            'timestamp':p.fields.timestamp_iso8601.0,
            'start':p.fields.start.0,
            'end':p.fields.end.0
            ])
            .collect(Collectors.toList());
"""
        },
        "logging": {
          "text": """
Anomalies:
==========
{{#ctx.payload._value}}
time={{timestamp}} 
airline={{airline}} 
score={{score}} (out of 100) 
responsetime={{actual}}ms (typical={{typical}}ms)
link= http://localhost:5601/app/ml#/timeseriesexplorer/?_g=(ml:(jobIds:!({{ctx.metadata.job_id}})),refreshInterval:(display:Off,pause:!f,value:0),time:(from:'{{start}}',mode:absolute,to:'{{end}}'))&_a=(filters:!(),mlSelectInterval:(interval:(display:Auto,val:auto)),mlSelectSeverity:(threshold:(display:warning,val:0)),mlTimeSeriesExplorer:(detectorIndex:0,entities:(airline:{{airline}})),query:(query_string:(analyze_wildcard:!t,query:'*')))
{{/ctx.payload._value}}
"""
        }
      }
    }
  }

You use "mustache" syntax to iterate through the results with the {{#name}} to start the loop and {{/name}} to end the loop definition

3

@richcollier, I tried with that, so let me show you more of my data. Its like I need mustache inside of mustache. ( Where I list brand.0 and partner.0 , I would like to know how many brands or partners there are, since this IS for one record.)

so, sometimes Brand={{brand.0}}
or
Brand={{brand.0}} {{brand.1}}

  "attachments": [
    {
      "color": "danger",
      "title": ":rotating_light: Alert -  Anomalies:",
      "text": """{{#ctx.payload.anomaly_details}} == *Anomaly on {{bucket_time}}:* ==
*Partner*={{partner.0}}           *Brand*={{brand.0}}
*Score*={{score}}                   *Actual*={{actual}}     *Typical*={{typical}}
< Dashboard Link >

      {{/ctx.payload.anomaly_details}}
       """

If I change my transform to this:

"transform": {
    "script": {
      "source": "def df = new DecimalFormat('##.##'); return [ 'anomaly_details':ctx.payload.hits.hits.stream().map(p -> ['start_time':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).minus(Duration.ofMinutes(20)).format(DateTimeFormatter.ofPattern('MMM d @ h:mm a')),'end_time':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).plus(Duration.ofMinutes(20)).format(DateTimeFormatter.ofPattern('MMM d @ h:mm a')),'bucket_time':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).format(DateTimeFormatter.ofPattern('MMM d @ h:mm a')),'job_id':p._source.job_id,'score':df.format(p._source.record_score),'actual':df.format(p._source.actual.0),'typical':df.format(p._source.typical.0),'partner':p._source.influencers[0]['influencer_field_values'],'brand':p._source.influencers[1]['influencer_field_values'],'start_time_iso_instant':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).minus(Duration.ofMinutes(20)).format(DateTimeFormatter.ISO_INSTANT),'end_time_iso_instant':Instant.ofEpochMilli(p._source.timestamp).atZone(ZoneId.systemDefault()).plus(Duration.ofMinutes(20)).format(DateTimeFormatter.ISO_INSTANT) ]).collect(Collectors.toList())]",
      "lang": "painless"
    }

In the above transform, the part Im talking about is:

'partner':p._source.influencers[0]['influencer_field_values'],'brand':p._source.influencers[1]['influencer_field_values']

As you can see I am using mustache already and the partner and brand are already inside the {{#ctx.payload.anomaly_details}}, but sometimes the partner or brand value can contain more than one value for the same doc. So how do I loop again inside them while inside the {{/ctx.payload.anomaly_details}} ?

4

Turns out, you can nest mustache.

Step 1: in your transform script, add the whole influencer_field_values objects to the map:

'partner':p._source.influencers[0]['influencer_field_values'],
'brand':p._source.influencers[1]['influencer_field_values']

Step 2: make your mustache nested

{{#ctx.payload.anomaly_details}}
== *Anomaly on {{bucket_time}}:* ==
*Partner*={{#partner}}{{.}} {{/partner}}           *Brand*={{#brand}}{{.}} {{/brand}}
*Score*={{score}}                   *Actual*={{actual}}     *Typical*={{typical}}
{{/ctx.payload.anomaly_details}}

Then the result would look something like:

== *Anomaly on Feb 1 @ 11:06 AM:* ==
*Partner*=p3 p4            *Brand*=b3 b4 
*Score*=14.24                   *Actual*=17     *Typical*=1.71
== *Anomaly on Feb 1 @ 10:00 AM:* ==
*Partner*=p1 p2            *Brand*=b1 b2 
*Score*=3.19                   *Actual*=17     *Typical*=1.71
5

@richcollier, Thank you so Much!! That worked perfectly!

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.