Hi. I am wondering if the following is possible using X-pack and kibana.
I have a large amount of system logs coming into elasticsearch trough logstash from different stacks.
I want to get the dashboard to display for example a notification in case of errors and and in that case be able to click to view each error log separately. I have written a watcher alarm as shown below.
{
"trigger": {
"schedule": {
"interval": "10s"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logstash*"
],
"types": [],
"body": {
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "error OR level:ERROR"
}
},
{
"range": {
"@timestamp": {
"gte": "now-10s"
}
}
}
]
}
},
"_source": [
"message"
],
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"log": {
"logging": {
"level": "info",
"text": "{{#ctx.payload.hits.hits}}{{_source.message}}{{/ctx.payload.hits.hits}}"
}
}
},
"throttle_period_in_millis": 9
}
which will show me the error logs, but I have trouble displaying the logs properly in the dashboard.
-In case of multiple hits, it will display all the messages in the same row
-This action will write the output in elasticsearch master node's logs. It is unnecessary.
I am wondering if the logging action is proper for this use case. Is there any other way to display the log messages in the dashboard? Also, is there a way to trigger these alarms checked via the dashboard, i.e. delete these alarms once confirmed?