Good day,
We've upgraded our subscription to Gold today but i still cannot see the Alerting option for Kibana.
I want to set up email alerts to notify me when certain keywords appear in the logs shipped by the Filebeat agent.
Your help would be greatly appreciated.
Good afternoon, Werner!
You'll need to confirm you're using Kibana 7.10 and have followed the setup and pre-requisites guide. Also, be sure you have a role and space that can access alerting.
I'm going to raise this with engineers with specific knowledge of filebeat, but if you can confirm the above, that would be helpful.
Hi there, thanks very much for the reply.
We are not using an on-premise Elastic stack. We are running it in Azure. Does it have to be on-premise for alerts to work?
Hi Werner,
Welcome to the community. 
No, you don't need to be on prem at all - everything should be available to you on the cloud, as long as you're running a version since 7.7 (though I'd recommend running the latest, 7.10).
That said, the specific alert you want won't be supported by the Kibana Alerting framework until the search alert is made available (you can track the progress here).
But I'm glad to say you should be able to use Watcher to achieve what you're looking for.
Have you given that a shot?
Hi Gidi and thank you 
I've given the Watcher a try, but I'm not 100% sure if this is correct:
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logs"
],
"rest_total_hits_as_int": true,
"body": {
"query": {
"bool": {
"must": [
{
"match": {
"message": "*websocket*"
}
}
],
"filter": [
{
"range": {
"logTimestamp": {
"gte": "now-7d",
"lt": "now"
}
}
}
]
}
},
"_source": [
"message"
],
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"send_email": {
"throttle_period_in_millis": 120000,
"email": {
"profile": "standard",
"to": [
"werner@blah.com"
],
"subject": "Kibana alert",
"body": {
"text": "Error message: {{ctx.payload.hits.hits.0._source.syslog_message}} "
}
}
},
"log": {
"logging": {
"level": "info",
"text": "{{ctx.payload.hits.total}} Errors have occured in the logs:{{#ctx.payload.hits.hits}}{{_id}}:{{/ctx.payload.hits.hits}}"
}
}
},
"throttle_period_in_millis": 900000
}
This is what I get when the watcher is executed:
{
"watch_id": "b5ec3c71-9263-4c2c-a848-ac5ac857d4dc",
"node": "Lmrgs_M5S4uWAAXFHAIR9g",
"state": "execution_not_needed",
"user": "werner",
"status": {
"state": {
"active": true,
"timestamp": "2020-11-30T11:33:21.384Z"
},
"last_checked": "2020-11-30T11:36:02.126Z",
"actions": {
"log": {
"ack": {
"timestamp": "2020-11-30T11:33:21.384Z",
"state": "awaits_successful_execution"
}
},
"send_email": {
"ack": {
"timestamp": "2020-11-30T11:33:21.384Z",
"state": "awaits_successful_execution"
}
}
},
"execution_state": "execution_not_needed",
"version": -1
},
"trigger_event": {
"type": "schedule",
"triggered_time": "2020-11-30T11:36:02.126Z",
"schedule": {
"scheduled_time": "2020-11-30T11:36:01.831Z"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logs"
],
"rest_total_hits_as_int": true,
"body": {
"query": {
"bool": {
"must": [
{
"match": {
"message": "*websocket*"
}
}
],
"filter": [
{
"range": {
"logTimestamp": {
"gte": "now-7d",
"lt": "now"
}
}
}
]
}
},
"_source": [
"message"
],
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"metadata": {
"name": "Test watch",
"xpack": {
"type": "json"
}
},
"result": {
"execution_time": "2020-11-30T11:36:02.126Z",
"execution_duration": 0,
"input": {
"type": "search",
"status": "success",
"payload": {
"_shards": {
"total": 0,
"failed": 0,
"successful": 0,
"skipped": 0
},
"hits": {
"hits": [],
"total": 0,
"max_score": 0
},
"took": 0,
"timed_out": false
},
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logs"
],
"rest_total_hits_as_int": true,
"body": {
"query": {
"bool": {
"must": [
{
"match": {
"message": "*exception*"
}
}
],
"filter": [
{
"range": {
"logTimestamp": {
"gte": "now-7d",
"lt": "now"
}
}
}
]
}
},
"_source": [
"message"
],
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
}
}
},
"condition": {
"type": "compare",
"status": "success",
"met": false,
"compare": {
"resolved_values": {
"ctx.payload.hits.total": 0
}
}
},
"actions": []
},
"messages": []
}
Sorry Werner, I'm not particularly familiar with Watcher, I just know it can be used in this manner.
I've pinged the team in charge of Watcher, hopefully they'll be able to help.
@werner.fletcher can you share the output of GET _cat/indices?s=index? It's fine if you'd like to filter that down to just the relevant indices.
Hi Joe,
Could you please explain how I would go about getting you that output? I'm very new to the whole Elastic stack thing so I'm not sure where to run that GET request for you?
Thank you.
How are you issuing the requests to define and execute watches? Those are probably HTTP PUT or POST requests, so use the same tooling (Kibana Dev Console, curl, etc) to issue a GET request. Here's a link to the docs for the 'cat indices API'.
I'm very new to the whole Elastic stack thing
Welcome! And in that case you are going to love the cat APIs, they're really excellent for exploring your Elasticsearch cluster.
Hi Joe,
Thanks for clarifying, I found the Console. Here is the output of the above request:
green open .apm-agent-configuration FRN0f6yiRrWcvu3OR6su0Q 1 0 0 0 283b 283b
green open .kibana_1 kO4gS38kTM2Qipt51Gniew 1 0 125 19 540.3kb 540.3kb
green open .kibana_task_manager_1 s_pO36b_TpSlP84MX5GBDA 1 0 2 2 11.1kb 11.1kb
green open .management-beats gMRn_o_BQfy8qAlY5Lwo1w 1 0 4 0 23.6kb 23.6kb
green open .security-7 PErx5STfR4uaSc72tvLdQQ 1 0 38 4 83kb 83kb
green open .slm-history-1-000005 EU0vD7QQRsCyDgF7qQ1bhw 1 0 2881 0 438.4kb 438.4kb
green open .slm-history-1-000006 hYA1zY6GSjaCGqeZEmr-lQ 1 0 2880 0 425kb 425kb
green open .slm-history-1-000007 pcK-PqtMRn24AADPRao1FA 1 0 2751 0 409.8kb 409.8kb
green open .slm-history-1-000008 2g6REX2JQdWBmEIx5MokLA 1 0 1990 0 318.8kb 318.8kb
green open .triggered_watches YR0qtDhbSGC13FvXDJMBcg 1 0 0 0 271b 271b
green open .watcher-history-10-2020.12.01 3wCL1OZhSX2pEE42NJGvwg 1 0 460 0 166.3kb 166.3kb
green open .watches hzqML9AlS9W4cF6Kos1tDg 1 0 2 1 20.1kb 20.1kb
green open apm-7.6.2-error-000001 X1_Lx4rLTFuywFXs7g_K6A 1 0 0 0 283b 283b
green open apm-7.6.2-error-000002 _3gSgdpUTTKP6DxJ7_TYFg 1 0 0 0 283b 283b
green open apm-7.6.2-error-000003 uvPyRsNTQIC9EbaNecmLkg 1 0 0 0 283b 283b
green open apm-7.6.2-error-000004 XfJhFP-XRH-OdILBb1pAow 1 0 4009 0 4.6mb 4.6mb
green open apm-7.6.2-error-000005 QytHjpufQD6uwD3L5ybPmw 1 0 8871 0 10.3mb 10.3mb
green open apm-7.6.2-error-000006 UoPs4wiERaG5u9_OQicxSw 1 0 0 0 283b 283b
green open apm-7.6.2-error-000007 4E6JYoDQT0Gcel6iQev5Og 1 0 0 0 283b 283b
green open apm-7.6.2-error-000008 1ZylsPjMRYeNTRP9ZjnJhw 1 0 0 0 283b 283b
green open apm-7.6.2-metric-000001 fqQNfm4bTOC39WdIgSD3qA 1 0 0 0 283b 283b
green open apm-7.6.2-metric-000002 3Z9zro31SdWzdghWBWiQ_g 1 0 0 0 283b 283b
green open apm-7.6.2-metric-000003 k_NLL3FuT2i7xqrfqiT1-g 1 0 0 0 283b 283b
green open apm-7.6.2-metric-000004 VC5NURaaQ1qvU8Pijsl8Og 1 0 35514 0 4mb 4mb
green open apm-7.6.2-metric-000005 W5iIheabRs-rSwku6nESzg 1 0 41378 0 4.6mb 4.6mb
green open apm-7.6.2-metric-000006 PLYfAC8DShyno7plDwh3rQ 1 0 0 0 283b 283b
green open apm-7.6.2-metric-000007 HJWGN9I5QGOITeVKGo1xlw 1 0 0 0 283b 283b
green open apm-7.6.2-metric-000008 SxRz5ebDRhWMUGqYJ2cyHA 1 0 0 0 283b 283b
green open apm-7.6.2-onboarding-2020.04.16 1NjQgjA6S42pq9X4bz6YSQ 1 0 1 0 6.7kb 6.7kb
green open apm-7.6.2-onboarding-2020.05.14 CU1ZXWY_TIiNNfrIR6TkSg 1 0 1 0 6.6kb 6.6kb
green open apm-7.6.2-onboarding-2020.06.30 njO8wW0RRXuqLLEftFxOCw 1 0 1 0 6.6kb 6.6kb
green open apm-7.6.2-profile-000001 GOhdXCJDRzKZEKEe3mkQzQ 1 0 0 0 283b 283b
green open apm-7.6.2-profile-000002 n_dQ5hABRpqZtM_pGVHVVQ 1 0 0 0 283b 283b
green open apm-7.6.2-profile-000003 Jk4bFY39SjGR-Fz3X3AWGw 1 0 0 0 283b 283b
green open apm-7.6.2-profile-000004 wEQn_vEJRYK6yWUptzTm2A 1 0 0 0 283b 283b
green open apm-7.6.2-profile-000005 Uv6-YwUJRveiePaM4ljRrg 1 0 0 0 283b 283b
green open apm-7.6.2-profile-000006 bh6sTM3rQhu879eW7dmNRA 1 0 0 0 283b 283b
green open apm-7.6.2-profile-000007 23HYUcN8S3e1DFzwGCjk6g 1 0 0 0 283b 283b
green open apm-7.6.2-profile-000008 02mRUJGlQ2SmPI5ODCUoOw 1 0 0 0 283b 283b
green open apm-7.6.2-span-000001 HIi6JvzoSpalcLESt-Fb8A 1 0 0 0 283b 283b
green open apm-7.6.2-span-000002 Pmg864f7Q1q9_LFf-1Z_aA 1 0 0 0 283b 283b
green open apm-7.6.2-span-000003 y4157x5qQjOG6NYjAl7BFQ 1 0 0 0 283b 283b
green open apm-7.6.2-span-000004 Pf069ksBSlGpBh8JCq-Wug 1 0 226 0 204.9kb 204.9kb
green open apm-7.6.2-span-000005 UqFxep8uSfe-FZPMHCSdjg 1 0 282 0 212.8kb 212.8kb
green open apm-7.6.2-span-000006 3NycqoaGQgueSEWqj9G7LQ 1 0 0 0 283b 283b
green open apm-7.6.2-span-000007 IQW5cRP0TaS3ArnrT-PSfg 1 0 0 0 283b 283b
green open apm-7.6.2-span-000008 TCyJyqkGSKycT8ZhluG_cA 1 0 0 0 283b 283b
green open apm-7.6.2-transaction-000001 AXaeKDZaTxKQTC_Ud9E7AA 1 0 0 0 283b 283b
green open apm-7.6.2-transaction-000002 gcuZI2nzSViQChUikp-pdg 1 0 0 0 283b 283b
green open apm-7.6.2-transaction-000003 4YK40QowSvake60iNeQ0gA 1 0 0 0 283b 283b
green open apm-7.6.2-transaction-000004 dg_11_QHRaagoNh52y61TQ 1 0 86 0 166kb 166kb
green open apm-7.6.2-transaction-000005 JVkVT5f4R5q2xnDCp63RcA 1 0 140 0 288.4kb 288.4kb
green open apm-7.6.2-transaction-000006 _R1U4TYzSPidzYvoUhFrSg 1 0 0 0 283b 283b
green open apm-7.6.2-transaction-000007 5SDgwL8JSxWRrswHbGu2Eg 1 0 0 0 283b 283b
green open apm-7.6.2-transaction-000008 8qVRWW8DQDu8y9EofJiRew 1 0 0 0 283b 283b
yellow open filebeat-7.10.0-2020.11.24-000001 Lgnl5iOYReyTgnUYRPRSww 1 1 768 0 245.5kb 245.5kb
yellow open filebeat-7.6.2-2020.04.16-000001 _iCavGT1RE2cpfXKs7DMHg 1 1 613 0 209.3kb 209.3kb
yellow open filebeat-7.6.2-2020.05.16-000002 q2-YiHZbSqGreqyEeUlBVA 1 1 6781 0 1.9mb 1.9mb
yellow open filebeat-7.6.2-2020.06.15-000003 oiiahuyATXeQADmQ4I0FGw 1 1 26888 0 7.8mb 7.8mb
yellow open filebeat-7.6.2-2020.07.15-000004 YSj0Mp07SmejK-GKjWHOeA 1 1 124759 0 38.9mb 38.9mb
yellow open filebeat-7.6.2-2020.08.14-000005 iQYwk7gLQjS2pNTchRNYOQ 1 1 146269 0 38.1mb 38.1mb
yellow open filebeat-7.6.2-2020.09.13-000006 BVC8nQOKSBSQTOGBPxDdgw 1 1 76588 0 15.3mb 15.3mb
yellow open filebeat-7.6.2-2020.10.13-000007 8W1KQ4HCQAi-PX2YycnA9g 1 1 74274 0 11.3mb 11.3mb
yellow open filebeat-7.6.2-2020.11.12-000008 2E6mstrtQpG7P992teOZWA 1 1 2886 0 866.6kb 866.6kb
yellow open filebeat-7.7.1-2020.06.18-000001 mFPID-vuTiSzwMJFsV1zFQ 1 1 79292 0 16mb 16mb
yellow open filebeat-7.7.1-2020.07.18-000002 Ew4jtn_jTba1IatmwwuE8g 1 1 0 0 283b 283b
yellow open filebeat-7.7.1-2020.08.17-000003 Ftystse7THCmSuPP5BW53w 1 1 0 0 283b 283b
yellow open filebeat-7.7.1-2020.09.16-000004 Ln2ej2eTT1O6UO3vQElCBA 1 1 0 0 283b 283b
yellow open filebeat-7.7.1-2020.10.16-000005 fHsDcMw9RympVbiEezVGMg 1 1 0 0 283b 283b
yellow open filebeat-7.7.1-2020.11.15-000006 ZfCxOVOnQsKjHqmPCe9fyg 1 1 0 0 283b 283b
yellow open filebeat-7.8.0-2020.06.23-000001 jSM9zoRiSqaHyGjw_iUO9A 1 1 1183285 0 232.4mb 232.4mb
yellow open filebeat-7.8.0-2020.07.23-000002 29z2QzCGSB2VoPCvlGf-KA 1 1 2974520 0 556.3mb 556.3mb
yellow open filebeat-7.8.0-2020.08.22-000003 dcBm4oZQTbOlg5RtfPCADg 1 1 8929825 0 1.3gb 1.3gb
yellow open filebeat-7.8.0-2020.09.21-000004 DyFcuA20SdSsyZzNi6cHmw 1 1 6572191 0 1.1gb 1.1gb
yellow open filebeat-7.8.0-2020.10.21-000005 qQA3-NBSR7uSYnkYjoJMsw 1 1 2923544 0 572.3mb 572.3mb
yellow open filebeat-7.8.0-2020.11.20-000006 _yp8U3jRTjuzLx8tpzRVnQ 1 1 1745155 0 356.7mb 356.7mb
yellow open heartbeat-7.8.1-2020.07.28-000001 vfQwUDbpS82zaP7PCKLnaQ 1 1 652627 0 163mb 163mb
yellow open heartbeat-7.8.1-2020.08.27-000002 MGvVhotsSkmLIfk4rZQ5zA 1 1 633959 0 158.5mb 158.5mb
yellow open heartbeat-7.8.1-2020.09.26-000003 DuwzwatzTnWNUAiWEGgrSA 1 1 620169 0 153.5mb 153.5mb
yellow open heartbeat-7.8.1-2020.10.26-000004 RPm45WE9SuywRHh1pRAxjw 1 1 626333 0 155.4mb 155.4mb
yellow open heartbeat-7.8.1-2020.11.25-000005 oeNedc8sRUai0nlTVbxJpQ 1 1 160114 0 42.9mb 42.9mb
green open ilm-history-1-000005 1pbO0sKJR4SkGumNdeAXDQ 1 0 200 0 106.3kb 106.3kb
green open ilm-history-1-000006 EiiA4aWcRnSVjzkWYlbZlw 1 0 199 0 98.2kb 98.2kb
green open ilm-history-1-000007 HU4Qc9-oQe6aowaUqRnDxg 1 0 199 0 97.8kb 97.8kb
green open ilm-history-1-000008 pj2pP32UTsiNSargh1Xakw 1 0 176 0 78.1kb 78.1kb
green open kibana_sample_data_logs cHVuAY41S3aw88UWfDogng 1 0 14074 0 11.2mb 11.2mb
Ah, okay, you've got "indices": [ "logs" ], but it doesn't look to me like you actually have an index called logs -- perhaps something like "indices": [ "filebeat-*" ], might be closer to what you're trying to do?
Hi Joe,
I think I finally got a solution. Here is the Watch in case anyone else ever needs something like this:
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"filebeat-*"
],
"rest_total_hits_as_int": true,
"body": {
"query": {
"bool": {
"must": [
{
"match_phrase": {
"message": "java heap space"
}
},
{
"match": {
"agent.hostname": "Server-1"
}
},
{
"range": {
"@timestamp": {
"gte": "now-5m"
}
}
}
]
}
},
"_source": [
"message",
"agent.hostname"
],
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 1
}
}
},
"actions": {
"send_email": {
"throttle_period_in_millis": 120000,
"email": {
"profile": "standard",
"to": [
"a@b.com"
],
"subject": "Kibana Alert",
"body": {
"text": "A Java Heap Space error has occurred"
}
}
},
"log": {
"logging": {
"level": "info",
"text": "A Java heap space error has occurred on server 'Server-1' {{ctx.payload.hits.total}} times"
}
}
},
"throttle_period_in_millis": 900000
}
Thanks for the assistance.
Great! I'm glad you've gotten things working. Cheers!
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.