Make the email body of my alert easier to read

Hello,

I have created an advanced watcher alert in order to monitor for each of my hosts which transactions have an http status code 500 to 600 . My alert code is the one below:

{
  "trigger": {
    "schedule": {
      "cron": [
        "0 * 0-1 * * ?",
        "0 * 3-22 * * ?",
        "0 15-59 23 * * ?",
        "0 58-59 2 * * ?"
      ]
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "apm-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must_not": {
                "term": {
                  "transaction.name": "TokenEndpoint#postAccessToken"
                }
              },
              "must": [
                {
                  "terms": {
                    "host.hostname": [
                      "sag-prd-cas-025.sag.services",
                      "sag-prd-cas-026.sag.services",
                      "sag-prd-cas-027.sag.services",
                      "sag-prd-cas-028.sag.services",
                      "sag-prd-cas-029.sag.services",
                      "sag-prd-cas-030.sag.services"
                    ]
                  }
                }
              ],
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m"
                    }
                  }
                },
                {
                  "range": {
                    "http.response.status_code": {
                      "gte": 500,
                      "lte": 600
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "hosts": {
              "terms": {
                "field": "host.hostname"
              },
              "aggs": {
                "transactions": {
                  "terms": {
                    "field": "transaction.name"
                  },
                  "aggs": {
                    "status": {
                      "terms": {
                        "field": "http.response.status_code"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 30
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "to": [
          "alexandros.ananikidis@sag-ag.ch"
        ],
        "subject": "5xx HTTP status code dedected",
        "body": {
          "text": "The Watcher has detected {{ctx.payload.hits.total}} times a 5xx HTTP status code during the last 1 minute.\n\n The detailed results are the following: \n\n {{ctx.payload.aggregations.hosts.buckets}} "
        }
      }
    }
  }
}

The alert works perfectly but the email body that i receive has a terrible structure for someone to read and understand where the http failures occure.

The email that i get has the following body:

The Watcher has detected 121 times a 5xx HTTP status code during the last 1 minute.

The detailed results are the following:

{0={doc_count=119, transactions={doc_count_error_upper_bound=0, sum_other_doc_count=0, buckets=[{doc_count=119, key=ArticlesController#getUpdatedAvailabilities, status={doc_count_error_upper_bound=0, sum_other_doc_count=0, buckets=[{doc_count=119, key=500}]}}]}, key=sag-prd-cas-029.sag.services}, 1={doc_count=2, transactions={doc_count_error_upper_bound=0, sum_other_doc_count=0, buckets=[{doc_count=1, key=ArticleSearchController#searchArticlesByCateIdsAndVehIds, status={doc_count_error_upper_bound=0, sum_other_doc_count=0, buckets=[{doc_count=1, key=500}]}}]}, key=sag-prd-cas-027.sag.services}}

What can i do in order to show that info in a more nice and clear way?

Thank you in advance

watcher allows you to make use of mustache templates to format data https://www.elastic.co/guide/en/x-pack/current/actions-email.html#configuring-email-actions
If you just need to convert data to a proper JSON format, you can leverage built-in formatter "{{#toJson}}ctx.payload{{/toJson}}"

Hello Mikhail,

Thank you for your asnwer.

I have put the command in my email body as you suggested and the code now is like this:

{
  "trigger": {
    "schedule": {
      "cron": [
        "0 * 0-1 * * ?",
        "0 * 3-22 * * ?",
        "0 15-59 23 * * ?",
        "0 58-59 2 * * ?"
      ]
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "apm-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must_not": {
                "term": {
                  "transaction.name": "TokenEndpoint#postAccessToken"
                }
              },
              "must": [
                {
                  "terms": {
                    "host.hostname": [
                      "sag-prd-cas-025.sag.services",
                      "sag-prd-cas-026.sag.services",
                      "sag-prd-cas-027.sag.services",
                      "sag-prd-cas-028.sag.services",
                      "sag-prd-cas-029.sag.services",
                      "sag-prd-cas-030.sag.services"
                    ]
                  }
                }
              ],
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m"
                    }
                  }
                },
                {
                  "range": {
                    "http.response.status_code": {
                      "gte": 200,
                      "lte": 600
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "hosts": {
              "terms": {
                "field": "host.hostname"
              },
              "aggs": {
                "transactions": {
                  "terms": {
                    "field": "transaction.name"
                  },
                  "aggs": {
                    "status": {
                      "terms": {
                        "field": "http.response.status_code"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 30
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "to": [
          "alexandros.ananikidis@sag-ag.ch"
        ],
        "subject": "5xx HTTP status code dedected",
        "body": {
          "text": "The Watcher has detected {{#toJson}}ctx.payload.hits.total{{/toJson}} times a 5xx HTTP status code during the last 1 minute.\n\n The detailed results are the following: \n\n {{ctx.payload.aggregations.hosts.buckets}} "
        }
      }
    }
  }
}

Nevertheless the output is still very difficult for someone to read as the image shows below:

I would like a solution that will provide an email body that someone will easily understand.
Is there a way?

Hi! What is output if you apply {{#toJson}} to the aggregations?

Hello
When i apply your change the code is like that

{
  "trigger": {
    "schedule": {
      "cron": [
        "0 * 0-1 * * ?",
        "0 * 3-22 * * ?",
        "0 15-59 23 * * ?",
        "0 58-59 2 * * ?"
      ]
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "apm-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must_not": {
                "term": {
                  "transaction.name": "TokenEndpoint#postAccessToken"
                }
              },
              "must": [
                {
                  "terms": {
                    "host.hostname": [
                      "sag-prd-cas-025.sag.services",
                      "sag-prd-cas-026.sag.services",
                      "sag-prd-cas-027.sag.services",
                      "sag-prd-cas-028.sag.services",
                      "sag-prd-cas-029.sag.services",
                      "sag-prd-cas-030.sag.services"
                    ]
                  }
                }
              ],
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m"
                    }
                  }
                },
                {
                  "range": {
                    "http.response.status_code": {
                      "gte": 200,
                      "lte": 600
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "hosts": {
              "terms": {
                "field": "host.hostname"
              },
              "aggs": {
                "transactions": {
                  "terms": {
                    "field": "transaction.name"
                  },
                  "aggs": {
                    "status": {
                      "terms": {
                        "field": "http.response.status_code"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 30
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "to": [
          "alexandros.ananikidis@sag-ag.ch"
        ],
        "subject": "5xx HTTP status code dedected",
        "body": {
          "text": "The Watcher has detected {{#toJson}}ctx.payload.hits.total{{/toJson}} times a 5xx HTTP status code during the last 1 minute.\n\n The detailed results are the following: \n\n {{#toJson}}ctx.payload.aggregations.hosts.buckets{{/toJson}} "
        }
      }
    }
  }
}

But as the image show nothing changes

Hello any news please?

You can loop through a list and only extract the elements that you are interested in, sth like

{#ctx.payload.aggregations.hosts.buckets}
Access field with in a bucket like this {{doc_count}}
{/ctx.payload.aggregations.hosts.buckets}

hope that helps as a start

Hello Alexander,

Thank you for the quick response.
According to your last recommendation it works when i use the folllowing command in my email body in order to retrieve my hostnames:
{{#ctx.payload.aggregations.hosts.buckets}}Host={{key}}\n\n{{/ctx.payload.aggregations.hosts.buckets}}

So i can extract the host name for each o my hosts.

Nevertheless, because i am using aggregation within aggregation when i want to extract the transactions names (per) host i dont get any results.
For that purpose i use the following command in my email body of my alert:

{{#ctx.payload.aggregations.hosts.buckets.transactions.buckets}}Transaction={{key}}\n\n{{/ctx.payload.aggregations.hosts.buckets.transactions.buckets}}.

The exact code that i use is the following:

  {
          "trigger": {
            "schedule": {
              "cron": [
                "0 * 0-1 * * ?",
                "0 * 3-22 * * ?",
                "0 15-59 23 * * ?",
                "0 58-59 2 * * ?"
              ]
            }
          },
          "input": {
            "search": {
              "request": {
                "search_type": "query_then_fetch",
                "indices": [
                  "apm-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                  "size": 0,
                  "query": {
                    "bool": {
                      "must_not": {
                        "term": {
                          "transaction.name": "TokenEndpoint#postAccessToken"
                        }
                      },
                      "must": [
                        {
                          "terms": {
                            "host.hostname": [
                              "sag-prd-cas-025.sag.services",
                              "sag-prd-cas-026.sag.services",
                              "sag-prd-cas-027.sag.services",
                              "sag-prd-cas-028.sag.services",
                              "sag-prd-cas-029.sag.services",
                              "sag-prd-cas-030.sag.services"
                            ]
                          }
                        }
                      ],
                      "filter": [
                        {
                          "range": {
                            "@timestamp": {
                              "gte": "now-1m"
                            }
                          }
                        },
                        {
                          "range": {
                            "http.response.status_code": {
                              "gte": 200,
                              "lte": 600
                            }
                          }
                        }
                      ]
                    }
                  },
                  "aggs": {
                    "hosts": {
                      "terms": {
                        "field": "host.hostname"
                      },
                      "aggs": {
                        "transactions": {
                          "terms": {
                            "field": "transaction.name"
                          },
                          "aggs": {
                            "status": {
                              "terms": {
                                "field": "http.response.status_code"
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          },
          "condition": {
            "compare": {
              "ctx.payload.hits.total": {
                "gt": 30
              }
            }
          },
          "actions": {
            "send_email": {
              "email": {
                "profile": "standard",
                "to": [
                  "alexandros.ananikidis@sag-ag.ch"
                ],
                "subject": "alex aggsss",
                "body": {
                  "text": " Hosts\n {{#ctx.payload.aggregations.hosts.buckets}}Host={{key}}\n\n{{/ctx.payload.aggregations.hosts.buckets}}\n\n Transactions per Host\n\n{{#ctx.payload.aggregations.hosts.buckets.transactions.buckets}}Transaction={{key}}\n\n{{/ctx.payload.aggregations.hosts.buckets.transactions.buckets}}"
                }
              }
            }
          }
        }

and a part of the result that i get from the dev tools in kibana when i run the alert is the following:

    "result" : {
      "execution_time" : "2020-04-15T15:52:37.252Z",
      "execution_duration" : 276,
      "input" : {
        "type" : "search",
        "status" : "success",
        "payload" : {
          "_shards" : {
            "total" : 379,
            "failed" : 0,
            "successful" : 379,
            "skipped" : 376
          },
          "hits" : {
            "hits" : [ ],
            "total" : 2715,
            "max_score" : null
          },
          "took" : 149,
          "timed_out" : false,
          "aggregations" : {
            "hosts" : {
              "doc_count_error_upper_bound" : 0,
              "sum_other_doc_count" : 0,
              "buckets" : [
                {
                  "doc_count" : 668,
                  "transactions" : {
                    "doc_count_error_upper_bound" : 0,
                    "sum_other_doc_count" : 171,
                    "buckets" : [
                      {
                        "doc_count" : 293,
                        "key" : "CheckTokenEndpoint#checkToken",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 293,
                              "key" : 200
                            }
                          ]
                        }
                      },
                      {
                        "doc_count" : 46,
                        "key" : "ResourceHttpRequestHandler",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 44,
                              "key" : 200
                            },
                            {
                              "doc_count" : 2,
                              "key" : 304
                            }
                          ]
                        }
                      },
                      {
                        "doc_count" : 33,
                        "key" : "DataAnalyticsController#track",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 33,
                              "key" : 200
                            }
                          ]
                        }
                      },
                      {
                        "doc_count" : 23,
                        "key" : "CartController#quickViewShoppingCart",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 23,
                              "key" : 200
                            }
                          ]
                        }
                      },
                      {
                        "doc_count" : 23,
                        "key" : "ContextController#updateEshopContext",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 23,
                              "key" : 200
                            }
                          ]
                        }
                      },
                      {
                        "doc_count" : 23,
                        "key" : "HaynesProController#getHaynesProLicense",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 15,
                              "key" : 404
                            },
                            {
                              "doc_count" : 8,
                              "key" : 200
                            }
                          ]
                        }
                      },
                      {
                        "doc_count" : 19,
                        "key" : "CartController#quickViewTotal",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 19,
                              "key" : 200
                            }
                          ]
                        }
                      },
                      {
                        "doc_count" : 13,
                        "key" : "MessageController#getUnAuthorizedMessages",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 13,
                              "key" : 200
                            }
                          ]
                        }
                      },
                      {
                        "doc_count" : 12,
                        "key" : "ArticlesController#getUpdatedAvailabilities",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 12,
                              "key" : 200
                            }
                          ]
                        }
                      },
                      {
                        "doc_count" : 12,
                        "key" : "HaynesProController#getHpAccessOptions",
                        "status" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 12,
                              "key" : 200
                            }
                          ]
                        }
                      }
                    ]
                  },
                  "key" : "sag-prd-cas-027.sag.services"
                }

and the email that i get is like that:

I've tested on my side with this Watch:

POST _xpack/watcher/watch/_execute
{
  "watch": {
    "trigger": {
      "schedule": {
        "daily": {
          "at": "noon"
        }
      }
    },
    "input": {
      "simple": {
        "_shards": {
          "total": 379,
          "failed": 0,
          "successful": 379,
          "skipped": 376
        },
        "hits": {
          "hits": [],
          "total": 2715,
          "max_score": null
        },
        "took": 149,
        "timed_out": false,
        "aggregations": {
          "hosts": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "doc_count": 668,
                "transactions": {
                  "doc_count_error_upper_bound": 0,
                  "sum_other_doc_count": 171,
                  "buckets": [
                    {
                      "doc_count": 293,
                      "key": "CheckTokenEndpoint#checkToken",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 293,
                            "key": 200
                          }
                        ]
                      }
                    },
                    {
                      "doc_count": 46,
                      "key": "ResourceHttpRequestHandler",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 44,
                            "key": 200
                          },
                          {
                            "doc_count": 2,
                            "key": 304
                          }
                        ]
                      }
                    },
                    {
                      "doc_count": 33,
                      "key": "DataAnalyticsController#track",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 33,
                            "key": 200
                          }
                        ]
                      }
                    },
                    {
                      "doc_count": 23,
                      "key": "CartController#quickViewShoppingCart",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 23,
                            "key": 200
                          }
                        ]
                      }
                    },
                    {
                      "doc_count": 23,
                      "key": "ContextController#updateEshopContext",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 23,
                            "key": 200
                          }
                        ]
                      }
                    },
                    {
                      "doc_count": 23,
                      "key": "HaynesProController#getHaynesProLicense",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 15,
                            "key": 404
                          },
                          {
                            "doc_count": 8,
                            "key": 200
                          }
                        ]
                      }
                    },
                    {
                      "doc_count": 19,
                      "key": "CartController#quickViewTotal",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 19,
                            "key": 200
                          }
                        ]
                      }
                    },
                    {
                      "doc_count": 13,
                      "key": "MessageController#getUnAuthorizedMessages",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 13,
                            "key": 200
                          }
                        ]
                      }
                    },
                    {
                      "doc_count": 12,
                      "key": "ArticlesController#getUpdatedAvailabilities",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 12,
                            "key": 200
                          }
                        ]
                      }
                    },
                    {
                      "doc_count": 12,
                      "key": "HaynesProController#getHpAccessOptions",
                      "status": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                          {
                            "doc_count": 12,
                            "key": 200
                          }
                        ]
                      }
                    }
                  ]
                },
                "key": "sag-prd-cas-027.sag.services"
              }
            ]
          }
        }
      }
    },
    "actions": {
      "logging": {
        "logging": {
          "text": """
Hosts
{{#ctx.payload.aggregations.hosts.buckets}}Host={{key}}

Transactions per Host

{{#transactions.buckets}}Transaction={{key}}
{{#status.buckets}}
- Status "{{key}}" - Count {{doc_count}}
{{/status.buckets}}

{{/transactions.buckets}}

{{/ctx.payload.aggregations.hosts.buckets}}
          """
        }
      }
    }
  }
}

Result:

      "actions" : [
        {
          "id" : "logging",
          "type" : "logging",
          "status" : "success",
          "logging" : {
            "logged_text" : """
Hosts
Host=sag-prd-cas-027.sag.services

Transactions per Host

Transaction=CheckTokenEndpoint#checkToken
- Status "200" - Count 293

Transaction=ResourceHttpRequestHandler
- Status "200" - Count 44
- Status "304" - Count 2

Transaction=DataAnalyticsController#track
- Status "200" - Count 33

Transaction=CartController#quickViewShoppingCart
- Status "200" - Count 23

Transaction=ContextController#updateEshopContext
- Status "200" - Count 23

Transaction=HaynesProController#getHaynesProLicense
- Status "404" - Count 15
- Status "200" - Count 8

Transaction=CartController#quickViewTotal
- Status "200" - Count 19

Transaction=MessageController#getUnAuthorizedMessages
- Status "200" - Count 13

Transaction=ArticlesController#getUpdatedAvailabilities
- Status "200" - Count 12

Transaction=HaynesProController#getHpAccessOptions
- Status "200" - Count 12



"""
          }
        }
      ]
1 Like

Thank you very much for the quick and precise solution.
Once again great help indeed. Much appreciated.

I also did an html formatting and according to that code:

 "body": {
    		  
   "html": """
 
<h2>Hosts:</h2>

 {{#ctx.payload.aggregations.hosts.buckets}}<h4>Host=({{key}})--------------------------</h4>
     <b>&ensp;-Transactions per Host:</b>
   <ul>
   {{#transactions.buckets}}<b>Transaction</b>={{key}}
     <dl>
   {{#status.buckets}}
     <li>&ensp;  <b>- Status</b> "{{key}}" <b>- Count</b> {{doc_count}} </li>
      {{/status.buckets}}
     </dl>
   {{/transactions.buckets}}
   </ul>
 {{/ctx.payload.aggregations.hosts.buckets}}         
		  """		
  
  }

and i retrieved the following very nice results:

Where can i learn and study more stuff about these useful mustache templates and html formatting in elastic?

because in elastic.co i believe only the basic operations are provided.

Best regards,
Alexandros

Mustache Templates are not Elastic specific. You can find examples even in stack overflow.
HTML formatting is just using HTML with mustache templates.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.