So, we use PagerDuty as an action for our alerts.
Unfortunately, with this action you can only set description
so we must try to plug as much information into the field before we fire off the alert. Sometimes this means that the field is more than 1024 characters and then gets rejected by Pagerduty:
"body": "{\"status\":\"invalid event\",\"message\":\"Event object is invalid\",\"errors\":[\"'summary' is too long (maximum is 1024 characters)\"]}"
I have been looking at trying to truncate that field to just send the first 950 characters, then ignore anything else, and stumbled across painless
and followed an example here:
"transform" : {
"script": {
"source" : "ctx.payload.hits.hits.0._source.log_message(e -> e.a.substring(0, (int) Math.min(e.a.length(), 1000)))",
"lang": "painless"
}
},
"actions": {
"notify-pagerduty" : {
"pagerduty" : {
"account": "my_pagerduty_account",
"description" : "X Console Error - {{ ctx.payload.hits.hits.0._source.agent.hostname }}/{{ ctx.payload.hits.hits.0._source.fields.environment }} - {{ ctx.payload.hits.hits.0._source.log_message }}",
"attach_payload" : true,
"contexts" : [
{
"type": "link",
"href": "X",
"text": "View the incident on {{ctx.payload.link}}"
}
]
}
},
So essentially, I would up with a partial log_message
which is fine at this stage as we would have enough information.
Unfortunately, i'm finding it hard to find much documentation on the painless
language, and to be honest, not really sure where I can begin.