Getting Alerts and Actions to work

[Ilai_Velocity (Ilai Fallach)]

Hey guys,
I got a few questions that I encountered when trying to setup alerts and actions.

We're on the Elasticsearch as a Service (Managed) solution.

Question 1 -
I'm following Defining alerts | Kibana Guide [7.x] | Elastic to define alerts, and I want to display a link back to the alert in the message. However, {{kibanaBaseUrl}} is empty.
Based on Configure Kibana | Kibana Guide [7.x] | Elastic it probably means that server.publicBaseUrl is not populated.

First, I would expect the managed elasticsearch to have this populated, and second, I tried editing it in the kibana.yml in the management console but I got an error: "Your changes cannot be applied. Kibana - 'server.publicBaseUrl': is not allowed".

Question 2 -
I tried placing newlines ("\n" ) inside a JSON field in the body of a webhook action, but it got messed up. How can create new lines in JSON messages?

Question 3 -
I'm trying to close an alert in Opsgenie after the alert was recovered, and this requires setting a variable URL (.../alerts/close/{{alertId}}) but I suspect mustache variables are not populated in webhook URLs. How can this be solved?

Question 4 -
Is there a way to troubleshoot why alerts are firing off?
I have an alert that is configured to alert for every unique value of a field, and it is firing for instance * (star), but that's not a value of the field. I suspect it's a bug on your side?

Thanks a lot in advance, expecting to hear from you soon so we can get this alerting solution to a "production-ready" level,
Ilai

[rashmi (kulkarni)]

cc @pmuellr /@gmmorris can you shed some light here please ?

Thanks

[gmmorris (Gidi Morris)]

hey @Ilai_Velocity ,
Sorry you encountered this somewhat broken UX

@Ilai_Velocity

First, I would expect the managed elasticsearch to have this populated

Agreed, and this is in process.
There are a variety of complications which might not be obvious looking in from the outside.
As there are a wide range of deployment strategies (such as multiple Kibana across managed and unmanaged, different proxies in front of served Kibana etc.), configuring this out ofthe box has proved tricky.

@Ilai_Velocity

I tried editing it in the kibana.yml in the management console but I got an error: "Your changes cannot be applied. Kibana - 'server.publicBaseUrl': is not allowed".

That said, this is an obvious oversight - you should be able to set this manually.
I've spoken to the team that owns this configuration and they have assured me they'll look to add this to the allowlist asap.

@Ilai_Velocity

Question 2 -
I tried placing newlines ("\n" ) inside a JSON field in the body of a webhook action, but it got messed up. How can create new lines in JSON messages?

I am aware of an issue with newlines which we're hoping to get to as part of 7.13 (though, I can't commit to that), but it looks like using explicit "\n" works.
Could you share the exact config that's failing for you?

@Ilai_Velocity

I'm trying to close an alert in Opsgenie after the alert was recovered, and this requires setting a variable URL (.../alerts/close/{{alertId}}) but I suspect mustache variables are not populated in webhook URLs. How can this be solved?

You're correct- this isn't supported at the moment.
We began looking into this and realised there's actually a security/safety difficulty here.
The complexity to this is that it would mean user A could provide authorization for a specific URL, but user B could then programatically change where that URL is pointing.
We have to find a good safe way of modeling this feature in a manner that keeps it secure and safe to use.

In the meantime, we're hoping to find capacity for the development of a dedicated OpsGenie Connector type in 7.13 (but as stated above, I can't commit to that).
You can track the issue here

@Ilai_Velocity

Is there a way to troubleshoot why alerts are firing off?

That depends on the Alert Type.

@Ilai_Velocity

I have an alert that is configured to alert for every unique value of a field, and it is firing for instance * (star), but that's not a value of the field. I suspect it's a bug on your side?

That does sound weird, but I can't confirm if it's a bug without more details.
Could you provide more details for where you're encountering this?

What Alert Type is this?
How is it configured?
Can you provide an example of the data that it's alerting on?

[Ilai_Velocity (Ilai Fallach)]

Hi Gidi! Thanks for the elaborate answers.

That said, this is an obvious oversight - you should be able to set this manually.
I've spoken to the team that owns this configuration and they have assured me they'll look to add this to the allowlist asap.

Thank you! Is there any estimation as to what ASAP means?

I am aware of an issue with newlines which we're hoping to get to as part of 7.13 (though, I can't commit to that), but it looks like using explicit "\n" works.
Could you share the exact config that's failing for you?

I am using a simple "\n", but I send it via the API and when I look at the webhook body in the UI, I see a new line, which is an invalid JSON message (and it also doesn't trigger, which brings me back to the question of how to debug the actions)

In the meantime, we're hoping to find capacity for the development of a dedicated OpsGenie Connector type in 7.13 (but as stated above, I can't commit to that).
You can track the issue here

Thank you.

That depends on the Alert Type.

Got it. So... "Inventory. Alert when the inventory exceeds a defined threshold."

That does sound weird, but I can't confirm if it's a bug without more details.
Could you provide more details for where you're encountering this?

What Alert Type is this?
How is it configured?
Can you provide an example of the data that it's alerting on?

So we figured out what it was: An instance of an alert of no data... I would say it's very unintuitive. Would consider changing how it's being presented.

Adding a question:

Question 5 -
I'm getting the following error:
An error occurred when decrypting the alert. Saved object [alert/<uuid>] not found

Indeed when I queried api/saved_objects/_find?type=alert there were 0 results. This looks like a bug?

Thanks again, Ilai