HTML sanitisation for Kibana link

[programagor (Jiří Bednář)]

Greetings

I'm implementing a spam detection tool using Watcher. Right now, I'm just going through the last 5 minutes of logs, and reporting any messages (grouped by the subject) which exceed some defined thresholds.
The message contains a list of all Subjects which exceeded the thresholds, and some extra detail. I would like the subjects to be links to Kibana, showing all entries relevant to the subject. This is my attempt (inside the watcher action → email → body → html):

Found events :
<ul>
    {{#ctx.vars.offenders}}
        <li>
            Subject: <a href="https://my-kibana/app/kibana#/discover?_g=(refreshInterval:(display:'30 seconds',pause:!f,section:1,value:30000),time:(mode:quick,from:now-12h,to:now))&_a=(columns:!(hdr_msgid,hdr_from,mailfrom,ip,geoip.country_name,geoip.city_name),index:edba35e0-8e7d-11e8-bdc5-a11e25f3d3ed,interval:m,query:(language:kuery,query:'hdr_subject:&quot;{{key}}&quot;'),sort:!('@timestamp',desc))">{{key}}</a> (Found {{doc_count}} times)
            <ul>
                {{#source_ips.buckets}}
                    <li>{{key}} (Found {{doc_count}} times)</li>
                {{/source_ips.buckets}}
            </ul>
        </li>
    {{/ctx.vars.offenders}}
</ul>
Details on search: <a href="https://my-kibana/app/kibana#/management/elasticsearch/watcher/watches/watch/ESRD/status">Kibana link</a>

However, this only works if the subject field is uncomplicated (i.e. no ampersands, quote marks, or non-UTF8 characters). Is there a way to HTML-sanitise the ctx.vars.offenders.key so that it can be injected into the Kibana URL? Or is there a better way to achieve this?

Also, is there a way for the "Details on search" link to point to the relevant event, instead of the entire history of this watcher?

[spinscale (Alexander Reelsen)]

I am not sure if this works, but it might be worth a try, you could try to URL encode that part using the mustache {{#url}} helper

[programagor (Jiří Bednář)]

Thanks, that almost solved it! Also, I confused HTML encoding with URL encoding, so I had to replace the " with %22.

Now the only problem I can see is the exclamation mark. I found that ! needs to be replaced with !! (or %21 with %21%21), otherwise Kibana throws an URL Parsing Error if there is an odd number of exclamation marks. Is there a tool to do this substitution (! → !!, no edge cases)?

[programagor (Jiří Bednář)]

Turns out, single quote marks also need to be escaped with a single exclamation mark.
I just did it using Painless:

for(int i=0;i<ctx.vars.offenders.size();i++)
{
    ctx.vars.offenders[i].keyEscaped=ctx.vars.offenders[i].key.replace("!","!!").replace("'","!'");
}

For reference, the whole link is now

<a href="https://my-kibana/app/kibana#/discover?_g=(refreshInterval:(display:'30 seconds',pause:!f,section:1,value:30000),time:(mode:quick,from:now-12h,to:now))&_a=(columns:!(hdr_msgid,hdr_from,mailfrom,ip,geoip.country_name,geoip.city_name),index:edba35e0-8e7d-11e8-bdc5-a11e25f3d3ed,interval:m,query:(language:kuery,query:'hdr_subject:%22{{#url}}{{keyEscaped}}{{/url}}%22'),sort:!('@timestamp',desc))">{{key}}</a>

Thanks for your help!

[programagor (Jiří Bednář)]

Additionally, double quote marks need to be escaped by backslash:

.replace("\"","\\\"")

Funnily enough, if you want to store this script in json, it becomes

"script": {
  "source": " ... .replace(\"\\\"\",\"\\\\\\\"\") ... "
  ...
}