Watcher that counts the documents that arrive to an index in kibana

Like this?

 "actions": {
    "email_1": {
      "email": {
        "profile": "standard",
        "attach_data": {
          "format": "yaml"
        },
        "to": [
          "victor.vera@megadvantage.com",
          "juan.jaramillo@megadvantage.com",
          "andres.molinac.pr@etb.com.co",
          "juan.ariasp1.pr@etb.com.co"
        ],
        "bcc": [
          "juancho.jaramillo16@gmail.com"
        ],
        "subject": "{{ctx.metadata.name}}",
        "body": {
          "text":  "Encountered {{ctx.payload.aggregations.heartbeat_count.value}} heartbeats in the last 1 day"
        }
      }
    }
  },

I just sent your text to slack looks like this and I see the number

Buen dia,
:warning:La herramienta de monitoreo a funcionado correctamente en las ultimas 24 horas 60474

Estado: UP:large_green_circle:
:clock1230:Hora:
Área: Networking

:bell:Mensaje Alerta: Por favor no responder a este mensaje

no good, nothing appears

 "actions": [
      {
        "id": "email_1",
        "type": "email",
        "status": "simulated",
        "email": {
          "message": {
            "id": "email_1__inlined__86e47b0a-e497-41e0-83bf-ff852ce4346b-2021-04-23T23:41:10.833199Z_26881",
            "sent_date": "2021-04-23T23:41:10.845017Z",
            "to": [
              "victor.vera@megadvantage.com",
              "juan.jaramillo@megadvantage.com",
              "andres.molinac.pr@etb.com.co",
              "juan.ariasp1.pr@etb.com.co"
            ],
            "bcc": [
              "juancho.jaramillo16@gmail.com"
            ],
            "subject": "Estado herramienta monitoreo",
            "body": {
              "text": "Encountered  heartbeats in the last 1 day"
            }
          }
        }
      }
    ]
  },
  "messages": []
}

what error is it? it doesn't work for me :frowning:

Something else is going on let me look close

My Slack :slight_smile:

I am sending you the whole code for your review:

{
  "trigger": {
    "schedule": {
      "daily": {
        "at": [
          "14:00"
        ]
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "heartbeat*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1d/d",
                      "lt": "now/d"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "heartbeat_count": {
              "value_count": {
                "field": "_index"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.heartbeat_count.value": {
        "gte": 1000
      }
    }
  },
  "actions": {
    "email_1": {
      "email": {
        "profile": "standard",
        "attach_data": {
          "format": "yaml"
        },
        "to": [
          "victor.vera@megadvantage.com",
          "juan.jaramillo@megadvantage.com",
          "andres.molinac.pr@etb.com.co",
          "juan.ariasp1.pr@etb.com.co"
        ],
        "bcc": [
          "juancho.jaramillo16@gmail.com"
        ],
        "subject": "{{ctx.metadata.name}}",
        "body": {
          "text":  "Encountered {{ctx.payload.aggregations.heartbeat_count.value}} heartbeats in the last 1 day"
        }
      }
    }
  },
  "transform": {
    "script": {
      "source": "return [ 'time_triggered': Instant.ofEpochMilli(ctx.trigger.triggered_time.getMillis()).atZone(ZoneId.of('America/Bogota')).format(DateTimeFormatter.ofPattern('YYYY-MM-dd HH:mm:ss')) ];",
      "lang": "painless"
    }
  }
}

To Test

  "trigger": {
    "schedule": {
      "interval": "15s"
    }
  },

To test take out that transform script at the bottom can try that later.

,
  "transform": {
    "script": {
      "source": "return [ 'time_triggered': Instant.ofEpochMilli(ctx.trigger.triggered_time.getMillis()).atZone(ZoneId.of('America/Bogota')).format(DateTimeFormatter.ofPattern('YYYY-MM-dd HH:mm:ss')) ];",
      "lang": "painless"
    }
  }

so just try this (put the simple email back in).
Email just to you (are you spamming your friends :slight_smile: )

 {
  "trigger": {
    "schedule": {
      "interval": "15s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "heartbeat-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1d/d",
                      "lt": "now/d"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "heartbeat_count": {
              "value_count": {
                "field": "_index"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.heartbeat_count.value": {
        "gte": 1000
      }
    }
  },
  "actions": {
    "notify-slack": {
      "slack": {
        "message": {
          "to": [
            "#stephenb-es-integration"
          ],
          "text": "{{ctx.metadata.name}} : Encountered {{ctx.payload.aggregations.heartbeat_count.value}} heartbeats in the last 1 day"
        }
      }
    }
  }
}

My output

[4:58]
test-heartbeat-watcher : Encountered 60474 heartbeats in the last 1 day

[4:58]
test-heartbeat-watcher : Encountered 60474 heartbeats in the last 1 day

[4:58]
test-heartbeat-watcher : Encountered 60474 heartbeats in the last 1 day

[4:59]
test-heartbeat-watcher : Encountered 60474 heartbeats in the last 1 day

@Juan_David_Jaramillo I need to step out for a bit... you are close....

Thank you, it was indeed because of the script that I did not get it.
But now I have a problem, I can't use the script? :frowning:

"actions": [
      {
        "id": "email_1",
        "type": "email",
        "status": "simulated",
        "email": {
          "message": {
            "id": "email_1__inlined__bf91ec99-e03d-4917-8f25-483335fc9da2-2021-04-24T00:04:05.187357Z_26886",
            "sent_date": "2021-04-24T00:04:05.201541Z",
            "to": [
              "victor.vera@megadvantage.com",
              "juan.jaramillo@megadvantage.com"
            ],
            "bcc": [
              "juancho.jaramillo16@gmail.com"
            ],
            "subject": "Estado herramienta monitoreo",
            "body": {
              "text": "Buen dia, \n\n ⚠La herramienta de monitoreo a funcionado correctamente en las ultimas  24 horas  \n \n there are 82076 documents in your index. Threshold is 1000.\n \n Estado: UP🟢 \n\n 🕧Hora:   \n\n Área: Networking \n \n🔔Mensaje Alerta: Por favor no responder a este mensaje \n\n "
            }
          }
        }
      }
    ]
  },

Yay ! :slight_smile:

The Script... Perhaps.... Probably and error in the script.... Perhaps I can take a look later.

What are you actually trying to do with the script?

Are You just trying to display the triggered time in the correct time zone?

yes,I need to display the actual time in my time zone.

Yes OK... I am sure we can figure that out.

Someone with painless might be faster... so perhaps a new thread..

Or I will take a look later...

ok, thanks, if it is important too, I need the real time because elasticsearch gives the wrong time, that's why I need the script.

Understood... it will be easy once we know the "Encantamiento mágico"

what those?

Sorry it means once we know the Magic Incantation... sometime we say that when we need to just know the exact syntax ... its kinda a joke / frustration about knowing the exact syntax ... and close does not work.

And here you go...

When you create that time variable it is part of ctx.payload

NOTE / EDIT this does not work as it writes over a payload.

  "actions": {
    "notify-slack": {
      "slack": {
        "message": {
          "to": [
            "#stephenb-es-integration"
          ],
          "text": "{{ctx.payload.time}} {{ctx.metadata.name}} : Encountered {{ctx.payload.aggregations.heartbeat_count.value}} heartbeats in the last 1 day"
        }
      }
    }
  },

  "transform" : {
    "script" : "return [ 'time' : ctx.trigger.triggered_time.withZoneSameInstant(ZoneId.of('America/Bogota')).format(DateTimeFormatter.ofPattern('YYYY-MM-dd HH:mm:ss')) ]" 
  }
}

NOTE I just changed to triggered_time since that is probable more accurate

output

APP [5:36 PM]
2021-04-23 19:36:28 test-heartbeat-watcher : Encountered heartbeats in the last 1 day

Ohh I just noticed that broke the payload again that must be what is happening!

Let me fix!!!

Ahhh I see it right here

A payload transform that executes a script on the current payload in the watch execution context and replaces it with a newly generated one. The following snippet shows how a simple script payload transform can be defined on the watch level:

A simple painless script that creates a new payload with a single time field holding the scheduled time.

So it is writing over our payload!!!

I will need to research ... I am sure it is simple...

ok, I hope it can be done

i'll figure it out but may not be tonight

And full circle... this may have been easier with kibana alerts :slight_smile:

After I figure this out I will do that

@Juan_David_Jaramillo

And here you go :slight_smile:

The Trick or (Magic Incantation) is that since the transform creates a new payload that overwrites the existing one, you need to save into the new payload the fields you want to use from the old payload ... and then reference them directly.

Note since I renamed / copied in the heartbeat_count.value into the new payload it is now at the top level of the new ctx.payload object

  "actions": {
    "notify-slack": {
      "slack": {
        "message": {
          "to": [
            "#stephenb-es-integration"
          ],
          "text": "{{ctx.metadata.name}} executed at {{ctx.payload.local_execution_time}} : Encountered {{ctx.payload.heartbeat_count}} heartbeats in the last 1 day"
        }
      }
    }
  },
  "transform": {
    "script": {
      "source": """
        return [
        'local_execution_time' : ctx.trigger.triggered_time.withZoneSameInstant(ZoneId.of('America/Bogota')).format(DateTimeFormatter.ofPattern('YYYY-MM-dd HH:mm:ss')),
        'heartbeat_count' : ctx.payload.aggregations.heartbeat_count.value
          ]
      """,
      "lang": "painless"
    }
  }
}

And the results ...

test-heartbeat-watcher executed at 2021-04-23 20:44:46 : Encountered 60480 heartbeats in the last 1 day

test-heartbeat-watcher executed at 2021-04-23 20:45:01 : Encountered 60480 heartbeats in the last 1 day

1 Like

Very Thanks!!! Amazing! :slight_smile:

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.