Watcher for http status code 500 in APM Transactions

[hsaqib (Saqib Hussain)]

Hi, I am trying to watch for every http request with code 500 and perform an action based on that. Does APM allow such watch when error is not explicitly fired from APM client but It logs that http request in Transactions section.

So far I've tried this config but this always fires no matter what

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "apm-*"
        ],
        "types": [],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "term": {
                    "context.service.name": "{{ctx.metadata.serviceName}}"
                  }
                },
                {
                  "term": {
                    "context.response.status_code": 500
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-{{ctx.metadata.timeRangeValue}}{{ctx.metadata.timeRangeUnit}}"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "error_groups": {
              "terms": {
                "min_doc_count": "{{ctx.metadata.threshold}}",
                "field": "error.grouping_key",
                "size": 10,
                "order": {
                  "_count": "desc"
                }
              },
              "aggs": {
                "sample": {
                  "top_hits": {
                    "_source": [
                      "error.log.message",
                      "error.exception.message",
                      "error.exception.handled",
                      "error.culprit",
                      "error.grouping_key",
                      "@timestamp"
                    ],
                    "sort": [
                      {
                        "@timestamp": "desc"
                      }
                    ],
                    "size": 1
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "always": {}
  },
  "actions": {
    "log_error": {
      "logging": {
        "level": "info",
        "text": "Your service \"{{ctx.metadata.serviceName}}\" has error groups which exceeds {{ctx.metadata.threshold}}"
      }
    },
    "ms_teams": {
      "webhook": {
        "scheme": "https",
        "host": "outlook.office.com",
        "port": 443,
        "method": "post",
        "path": "/webhook/xxx/xxx",
        "params": {},
        "headers": {
          "Content-Type": "application/json"
        },
        "body": "__json__::{\"text\":\"Your service <b>\\\"{{ctx.metadata.serviceName}}\\\"</b> has error groups which exceeds {{ctx.metadata.threshold}} occurrences within \\\"{{ctx.metadata.timeRangeValue}}{{ctx.metadata.timeRangeUnit}}\\\"\\n{{#ctx.payload.aggregations.error_groups.buckets}}\"}"
      }
    }
  },
  "metadata": {
    "emails": [
      "xxxx@gmail.com"
    ],
    "timeRangeValue": 1,
    "threshold": 1,
    "trigger": "This value must be changed in trigger section",
    "serviceName": "node-server-live",
    "timeRangeUnit": "m"
  }
}```

[sqren (Søren Louv Jansen)]

Hi,

Yes, you can create watches on transaction documents (and any other document for that matter).
Since it is a transaction document, please note that it does not have any of the error properties, like error.grouping_key, error.log.message, error.exception.message etc.

I've replaced the error.grouping_key with transaction.name and added context.request.url.full and context.request.method to the aggregation output, since they might be useful in your watch output.

{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "context.service.name": "{{ctx.metadata.serviceName}}"
          }
        },
        {
          "term": {
            "context.response.status_code": 500
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-{{ctx.metadata.timeRangeValue}}{{ctx.metadata.timeRangeUnit}}"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "transaction_groups": {
      "terms": {
        "min_doc_count": "{{ctx.metadata.threshold}}",
        "field": "transaction.name",
        "size": 10,
        "order": {
          "_count": "desc"
        }
      },
      "aggs": {
        "sample": {
          "top_hits": {
            "_source": [
              "context.request.url.full",
              "context.request.method",
              "@timestamp"
            ],
            "sort": [
              {
                "@timestamp": "desc"
              }
            ],
            "size": 1
          }
        }
      }
    }
  }
}

If you want to test out the above, refer to the Kibana Dev Tools and run the query like:

GET apm-*/_search
{
  "size": 0,
  "query": { ... },
  "aggs": { ... }
}