# How can i get separate email for distinct services in logs by single watcher
**URL:** https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting
**Created:** [March 20, 2019, 12:24pm UTC](https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136 "2019-03-20T12:24:08Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@akpandy](https://discuss.elastic.co/u/akpandy)
#### Post date: [March 20, 2019, 12:24pm UTC](https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136/1 "2019-03-20T12:24:08Z")
Hi,
I am new to ELK, I am having single log file where all service failure gets logged.
Can i create a single watcher to generate multiple alert for each service failure.
For example: If i have failure for serviceA, serviceB, serviceC in log then my watcher should generate 3 email having their corresponding failure details.
Kindly suggest
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [March 20, 2019, 1:13pm UTC](https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136/2 "2019-03-20T13:13:23Z")
Currently, each alert can only sent out a single email. If you need to send out separate emails, the easiest part might be to create an alert per service.
Hope this helps!
--Alex
---
### Author:  [@akpandy](https://discuss.elastic.co/u/akpandy)
#### Post date: [March 20, 2019, 2:51pm UTC](https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136/3 "2019-03-20T14:51:11Z")
Thanks a lot for reply,
In our current architecture, we are sending emails in case of service failure with error details to Service now for incident creation.
We have around 400 services so i have to create 400 watcher alert to achieve the same...
Is there any way or work around to achieve the same.
Thanks
Abhishek
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [March 20, 2019, 3:11pm UTC](https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136/4 "2019-03-20T15:11:24Z")
you could work around that by sending an alert via HTTP to logstash (using the logstash [HTTP input](https://www.elastic.co/guide/en/logstash/6.6/plugins-inputs-http.html)), and then use logstash to send out several emails, by splitting the incoming event using the the [split filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html) and then send emails using the [email output](https://www.elastic.co/guide/en/logstash/6.6/plugins-outputs-email.html)
Hope this helps!
--Alex
---
### Author:  [@akpandy](https://discuss.elastic.co/u/akpandy)
#### Post date: [March 21, 2019, 4:39pm UTC](https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136/5 "2019-03-21T16:39:17Z")
Thanks,
I tried to do the work around solution as you suggested,
I updated the watcher and trying to send payload to logstash via webhook, but getting error like "path not found", Please find below the simulation result for the exceute action below. Kindly suggest. logstash http input plug in is installed.
{
"watch\_id": "_inlined_",
"node": "zJbMhvCbS3aqrVrBBFHWJg",
"state": "executed",
"user": "elastic",
"status": {
"state": {
"active": true,
"timestamp": "2019-03-21T15:40:55.422Z"
},
"last\_checked": "2019-03-21T15:40:55.423Z",
"last\_met\_condition": "2019-03-21T15:40:55.423Z",
"actions": {
"my\_webhook": {
"ack": {
"timestamp": "2019-03-21T15:40:55.422Z",
"state": "awaits\_successful\_execution"
},
"last\_execution": {
"timestamp": "2019-03-21T15:40:55.423Z",
"successful": false,
"reason": "received [404] status code"
}
}
},
"execution\_state": "executed",
"version": -1
},
"trigger\_event": {
"type": "manual",
"triggered\_time": "2019-03-21T15:40:55.423Z",
"manual": {
"schedule": {
"scheduled\_time": "2019-03-21T15:40:55.423Z"
}
}
},
"input": {
"search": {
"request": {
"search\_type": "query\_then\_fetch",
"indices": [
"pq-icelog\*"
],
"types": ,
"body": {
"size": 20,
"query": {
"bool": {
"must": [
{
"match": {
"eventDetails.eventType": "ERROR"
}
},
{
"range": {
"@timestamp": {
"gte": "now-2d",
"lt": "now"
}
}
}
]
}
},
"aggs": {
"group\_by\_serviceName": {
"terms": {
"field": "interfaceHeader.className.keyword",
"size": 5
},
"aggs": {
"group\_by\_logLevel": {
"terms": {
"field": "eventDetails.eventType.keyword",
"size": 5
},
"aggs": {
"get\_latest": {
"terms": {
"field": "@timestamp",
"size": 1,
"order": {
"\_key": "desc"
}
}
}
}
}
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 0
}
}
},
"metadata": {
"name": "Logstash\_Webhook\_Alert",
"xpack": {
"type": "json"
}
},
"result": {
"execution\_time": "2019-03-21T15:40:55.423Z",
"execution\_duration": 26,
"input": {
"type": "search",
"status": "success",
"payload": {
"\_shards": {
"total": 195,
"failed": 0,
"successful": 195,
"skipped": 180
},
"hits": {
"hits": ,
"total": 0,
"max\_score": null
},
"took": 19,
"timed\_out": false,
"aggregations": {
"group\_by\_serviceName": {
"doc\_count\_error\_upper\_bound": 0,
"sum\_other\_doc\_count": 0,
"buckets":
}
}
},
"search": {
"request": {
"search\_type": "query\_then\_fetch",
"indices": [
"pq-icelog\*"
],
"types": ,
"body": {
"size": 20,
"query": {
"bool": {
"must": [
{
"match": {
"eventDetails.eventType": "ERROR"
}
},
{
"range": {
"@timestamp": {
"gte": "now-2d",
"lt": "now"
}
}
}
]
}
},
"aggs": {
"group\_by\_serviceName": {
"terms": {
"field": "interfaceHeader.className.keyword",
"size": 5
},
"aggs": {
"group\_by\_logLevel": {
"terms": {
"field": "eventDetails.eventType.keyword",
"size": 5
},
"aggs": {
"get\_latest": {
"terms": {
"field": "@timestamp",
"size": 1,
"order": {
"\_key": "desc"
}
}
}
}
}
}
}
}
}
}
}
},
"condition": {
"type": "compare",
"status": "success",
"met": true,
"compare": {
"resolved\_values": {
"ctx.payload.hits.total": 0
}
}
},
"actions": [
{
"id": "my\_webhook",
"type": "webhook",
"status": "failure",
"transform": {
"type": "script",
"status": "success",
"payload": {
"hits": ,
"total": 0,
"max\_score": null
}
},
"reason": "received [404] status code",
"webhook": {
"request": {
"host": "10.132.1.2",
"port": 9615,
"scheme": "http",
"method": "post",
"path": "testAlert.json",
"headers": {
"Content-type": "application/json"
},
"body": "{hits=, total=0, max\_score=null}"
},
"response": {
"status": 404,
"headers": {
"content-length": [
"71"
],
"content-type": [
"application/json"
],
"x-content-type-options": [
"nosniff"
],
"x-cascade": [
"pass"
]
},
"body": "{"path":"/testAlert.json","status":404,"error":{"message":"Not Found"}}"
}
}
}
]
},
"messages":
}
---
### Author:  [@akpandy](https://discuss.elastic.co/u/akpandy)
#### Post date: [March 22, 2019, 11:21am UTC](https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136/6 "2019-03-22T11:21:14Z")
Addition to the above, when i executed the below i received error
bin/logstash -e "input { http { } } output { stdout { codec =\> rubydebug} }"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-22T10:50:00,562][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
ERROR: Configuration reloading can't be used with 'config.string' (-e).
usage:
bin/logstash -f CONFIG\_PATH [-t] [-r] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE\_NAME [-M "MODULE\_NAME.var.PLUGIN\_TYPE.PLUGIN\_NAME.VARIABLE\_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG\_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[2019-03-22T10:50:00,586][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [March 22, 2019, 12:35pm UTC](https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136/7 "2019-03-22T12:35:32Z")
the default port is 8080 according to the logstash http input documentation at [https://www.elastic.co/guide/en/logstash/6.6/plugins-inputs-http.html](https://www.elastic.co/guide/en/logstash/6.6/plugins-inputs-http.html)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 19, 2019, 12:38pm UTC](https://discuss.elastic.co/t/how-can-i-get-separate-email-for-distinct-services-in-logs-by-single-watcher/173136/8 "2019-04-19T12:38:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.