{#toJson} not escaping characters properly in Watcher webhook action

[CommieBobDole]

I am sending my Cisco ASA logs to Logstash, and am attempting to write a watch that will, when an invalid login is detected, perform a webhook action to post JSON containing some pertinent information about the alert as well as the source log messages that matched the query to a third-party listener.

The problem I'm running into is that I can't seem to get it to post ctx.payload.hits.hits as valid JSON - if I use the mustache template {{ctx.payload.hits.hits}}, I get keys and values without the right quotes for JSON, but if use {#toJson}ctx.payload.hits.hits{{/toJson}}, I get otherwise valid JSON in which some double quotes are not properly escaped, resulting in invalid JSON. Example excerpts are below:

When querying the watcher history, the "message" portion of the hit appears as below:

ASA-6-605004: Login denied from 10.42.42.104/6001 to inside:10.42.42.1/https for user \"*****\"\n

{{ctx.payload.hits.hits}} posts the same "message", but without the backslash escapes on the double quotes, as expected since it's not JSON formatted:

%ASA-6-605004: Login denied from 10.42.42.104/6545 to inside:10.42.42.1/https for user "*****"\n

But if I use {{#toJson}}ctx.payload.hits.hits{{/toJson}}, it posts a body which is properly JSON formatted with the exception of the message:

%ASA-6-605004: Login denied from 10.42.42.104/6559 to inside:10.42.42.1/https for user \\"*****\\"\\n

The last part should correctly be \\"\\" to properly escape both the backslash and the double quotes, or it should remove the original backslashes entirely (since they're just there to escape the quotes anyway) and render it as "" .

Am I doing something wrong here? I've gone at this from a couple of different directions, but it seems to come back to the text being escaped improperly by the built-in function. Any ideas for a workaround?

Thank you in advance for any help - I have added the full Watch below. I can provide the result from the watch history as well if needed (too many chars to fit in this post)

Eric

## Watch:

PUT _xpack/watcher/watch/cisco_failed_login
{
  "trigger": {
    "schedule": {
      "interval": "15s"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          "logstash-*"
        ],
        "body": {
          "min_score": 2,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gt": "now-15s"
                    }
                  }
                },
                {
                  "match": {
                    "ciscotag": "ASA-6-605004"
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
   "actions": {
    "test_webhook": {
      "webhook": {
        "method": "POST",
        "host": "10.42.42.18",
        "port": 80,
        "path": "/test",
        "headers": {
          "Content-Type": "application/json"
        },
        "body": "{\"{{ctx.watch_id}}\" : \"{{ctx.payload.hits.total}}\",\"Base Events\": {{#toJson}}ctx.payload.hits.hits{{/toJson}}}"
      }
    }
  }
}

[spinscale (Alexander Reelsen)]

Hey,

couple of things here:

a) It is going to be very hard if y our JSON keys are not stable (i.e. by using the id as the key value) if you want to find it again.
b) Also I would refrain from using spaces in key names - you never know which scripting language tries to access data and fails
c) When you use the toJson wrapper it is easier to wrap the whole object in it instead of only parts of it

In order to do c) you can use a transform, like in this example

POST _xpack/watcher/watch/_execute
{
  "watch": {
    "trigger": {
      "schedule": {
        "interval": "1m"
      }
    },
    "input": {
      "simple": {}
    },
    "actions": {
      "test_webhook": {
        "transform" : {
          "script" : {
            "inline" : "def x = [:]; x.put(ctx.watch_id, ctx.payload.hits.total) ; x.put('Base Events', ctx.payload.hits.hits) ; return x"
          }
        },
        "webhook": {
          "connection_timeout" : "1s",
          "read_timeout" : "1s",
          "url" : "http://127.0.0.1:9200/foo/bar/1",
          "method": "PUT",
          "headers": {
            "Content-Type": "application/json"
          },
          "body": "{{#toJson}}ctx.payload{{/toJson}}}"
        }
      }
    }
  },
  "alternative_input": {
    "took": 4,
    "timed_out": false,
    "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
    },
    "hits": {
      "total": 1,
      "max_score": 1,
      "hits": [
        {
          "_index": "foo",
          "_type": "bar",
          "_id": "1",
          "_score": 1,
          "_source": {
            "message": "ASA-6-605004: Login denied from 10.42.42.104/6001 to inside:10.42.42.1/https for user \"*****\"\n"
          }
        }
      ]
    }
  }
}

For further questions, please also use the execute watch API like above, as it makes it much simpler to reproduce your problem. Thanks!

Hope this helps

--Alex

[spinscale (Alexander Reelsen)]

Hey,

I got it to fail as well now, investigating... I will come back once I know more

--Alex

[spinscale (Alexander Reelsen)]

Hey,

turns out that I had one } too much in this line, make sure it only ends with two }}

          "body": "{{#toJson}}ctx.payload{{/toJson}}"

[CommieBobDole]

Alex,

So I tried this, and continued to get the same problems with the posted output. After some further investigation it appears that the problem is on the receiving end - I was using mod_dumpio on the remote server to verify the correct payload was being posted, and the server or some other component is actually escaping the backslashes after the (correct) JSON is received, which I verified with a packet capture.

Thank you very much for your help - building the payload via script then JSON encoding it all at once is a much more elegant way to do things, and the tested-good method you provided let me realize that the problem had to be elsewhere.

Thanks,

Eric