How to make alarm dashboard on standard system log errors

[rottis]

Hi. I am wondering if the following is possible using X-pack and kibana.
I have a large amount of system logs coming into elasticsearch trough logstash from different stacks.
I want to get the dashboard to display for example a notification in case of errors and and in that case be able to click to view each error log separately. I have written a watcher alarm as shown below.

{
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "logstash*"
        ],
        "types": [],
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "query_string": {
                    "query": "error OR level:ERROR"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-10s"
                    }
                  }
                }
              ]
            }
          },
          "_source": [
            "message"
          ],
          "sort": [
            {
              "@timestamp": {
                "order": "desc"
              }
            }
          ]
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "log": {
      "logging": {
        "level": "info",
        "text": "{{#ctx.payload.hits.hits}}{{_source.message}}{{/ctx.payload.hits.hits}}"
      }
    }
  },
  "throttle_period_in_millis": 9
}

which will show me the error logs, but I have trouble displaying the logs properly in the dashboard.
-In case of multiple hits, it will display all the messages in the same row
-This action will write the output in elasticsearch master node's logs. It is unnecessary.
I am wondering if the logging action is proper for this use case. Is there any other way to display the log messages in the dashboard? Also, is there a way to trigger these alarms checked via the dashboard, i.e. delete these alarms once confirmed?

[spinscale (Alexander Reelsen)]

Hey,

consider the logging for debugging purposes, but not for a real production use case. It's much better to use one of the other outputs like sending a message to slack/hipchat - in which you could provide a link to a dashboard that contains information about errors.

--Alex