Action - Sending Email Alert through Gmail or Outlook

Sanju_Mon · October 11, 2017, 10:59am

Hi,
In my setup, there is an Elasticsearch running which has XS logs being pumped in it through Broadworks. X-Pack is also installed. My requirement is to send and email alert in case of a specific condition. For this purpose , I have added the following in /etc/elasticsearch/elasticsearch.yml

xpack.notification.email.account:
    gmail_account:
        profile: gmail
        smtp:
            auth: true
            starttls.enable: true
            host: smtp.gmail.com
            port: 587
            user: <username>
            password: <password>

Similarly, I am creating a watcher using curl command as below:

curl -u elastic:changeme -XPUT '<ip_address>:9200/_xpack/watcher/watch/log_error_watch?pretty' -H 'Content-Type: application/json' -d'
{
  "trigger" : { "schedule" : { "interval" : "5s" }},
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "bwlog*" ],
        "body" : {
          "query" : {
            "match" : { "sipmethod" : "REGISTER" }
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : { "ctx.payload.hits.total" : { "gt" : 5 }}
  },
  "actions" : {
    "send_email" : {
      "email" : {
	    "to" : "<recipient_address>",
	    "subject" : "Watcher Notification",
            "body" : "Found {{ctx.payload.hits.total}} errors in the logs"
      }
    }
  }
}
'

After restarting the elasticsearch with new config, I have created the watcher and logs are also pumped into it. I can view the incoming logs in Kibana.

But I don't know, where the problem is stuck so that the mail alert is not working. For testing purpose, I even provided the access for less secure apps of my gmail account in gmail account settings. Still, the mail is not getting triggered. I tried all possibilities as far as I learnt so far.

Please help me through this to get the email action part done.
Thanks in advance

richcollier · October 11, 2017, 7:49pm

Two simple things to test first:

Does your watch really fire when expected? You can add the logging action to see the output of the watch is matching what/when you expect.
Ensure that the machine executing the watch can communicate with smtp.gmail.com. Simply validate that you get the following connection response when telneting to port 587:

%> telnet smtp.gmail.com 587
Trying 173.194.205.108...
Connected to gmail-smtp-msa.l.google.com.
Escape character is '^]'.
220 smtp.gmail.com ESMTP b57sm8719026qtb.3 - gsmtp

Sanju_Mon · October 17, 2017, 5:31am

Got the telnet part done. Thanks @richcollier

Arun_Raj · October 20, 2017, 7:01am

Hi Team

I 've configured a job in x-pack to monitor the real time data, ie: if there is any anomaly found in my data then the system should trigger an email to my box. But the system is failing to send email.

I tried the below

Watch created using curl command from command line – This gets triggered and the mails were sent.

"last_met_condition" : "2017-10-20T03:22:21.639Z",
"actions" : {
"send_email" : {
"ack" : {
"timestamp" : "2017-10-20T03:21:51.626Z",
"state" : "ackable"
},
"last_execution" : {
"timestamp" : "2017-10-20T03:22:16.636Z",
"successful" : true
},
"last_successful_execution" : {
"timestamp" : "2017-10-20T03:22:16.636Z",
"successful" : true
}
watch created when the ML job was created from GUI - email is not getting triggered

for this case
"send_email" : {
"ack" : {
"timestamp" : "2017-10-20T01:07:20.937Z",
"state" : "awaits_successful_execution"
}
}

In the above, action state is in awaits_successful_execution but I could see some anomalies in the data passed.

I did force execution via curl -XPOST 'x.x.x.x:9200/_xpack/watcher/watch/ml-test/_execute?pretty' -H 'Content-Type: application/json' -d'
{
"record_execution" : true
}
'

but the status didn't change

any kind of help would be appreciated

Thanks
Arun

Arun_Raj · October 23, 2017, 4:47am

any help pl

spinscale · October 23, 2017, 7:40am

please paste the full output of the execute watch API here.

Thanks!

Arun_Raj · October 23, 2017, 9:00am

{
"_id" : "ml-s2_e6d68302-8d1b-4473-a0dc-aaa82ab06fbd-2017-10-23T08:55:04.407Z",
"watch_record" : {
"watch_id" : "ml-s2",
"state" : "execution_not_needed",
"status" : {
"state" : {
"active" : false,
"timestamp" : "2017-10-20T08:56:15.979Z"
},
"last_checked" : "2017-10-23T08:55:04.407Z",
"actions" : {
"log" : {
"ack" : {
"timestamp" : "2017-10-20T01:07:20.937Z",
"state" : "awaits_successful_execution"
}
},
"send_email" : {
"ack" : {
"timestamp" : "2017-10-20T01:07:20.937Z",
"state" : "awaits_successful_execution"
}
}
}
},
"trigger_event" : {
"type" : "manual",
"triggered_time" : "2017-10-23T08:55:04.407Z",
"manual" : {
"schedule" : {
"scheduled_time" : "2017-10-23T08:55:04.407Z"
}
}
},
"input" : {
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
".ml-anomalies-*"
],
"types" : [ ],
"body" : {
"size" : 0,
"query" : {
"bool" : {
"filter" : [
{
"term" : {
"job_id" : "s2"
}
},
{
"range" : {
"timestamp" : {
"gte" : "now-10m"
}
}
},
{
"terms" : {
"result_type" : [
"bucket",
"record",
"influencer"
]
}
}
]
}
},
"aggs" : {
"bucket_results" : {
"filter" : {
"range" : {
"anomaly_score" : {
"gte" : 25
}
}
},
"aggs" : {
"top_bucket_hits" : {
"top_hits" : {
"sort" : [
{
"anomaly_score" : {
"order" : "desc"
}
}
],
"_source" : {
"includes" : [
"job_id",
"result_type",
"timestamp",
"anomaly_score",
"is_interim"
]
},

Arun_Raj · October 23, 2017, 9:01am

"size" : 1,
"script_fields" : {
"start" : {
"script" : {
"lang" : "painless",
"inline" : "new Date(doc["timestamp"].date.getMillis()-doc["bucket_span"].value * 1000 * params.padding)",
"params" : {
"padding" : 10
}
}
},
"end" : {
"script" : {
"lang" : "painless",
"inline" : "new Date(doc["timestamp"].date.getMillis()+doc["bucket_span"].value * 1000 * params.padding)",
"params" : {
"padding" : 10
}
}
},
"timestamp_epoch" : {
"script" : {
"lang" : "painless",
"inline" : "doc["timestamp"].date.getMillis()/1000"
}
},
"timestamp_iso8601" : {
"script" : {
"lang" : "painless",
"inline" : "doc["timestamp"].date"
}
},
"score" : {
"script" : {
"lang" : "painless",
"inline" : "Math.round(doc["anomaly_score"].value)"
}
}
}
}
}
}
},
"influencer_results" : {
"filter" : {
"range" : {
"influencer_score" : {
"gte" : 3
}
}
},
"aggs" : {
"top_influencer_hits" : {
"top_hits" : {
"sort" : [
{
"influencer_score" : {
"order" : "desc"
}
}
],
"_source" : {
"includes" : [
"result_type",
"timestamp",
"influencer_field_name",
"influencer_field_value",
"influencer_score",
"isInterim"
]
},
"size" : 3,
"script_fields" : {
"score" : {
"script" : {
"lang" : "painless",
"inline" : "Math.round(doc["influencer_score"].value)"
}
}
}
}
}
}
},
"record_results" : {
"filter" : {
"range" : {
"record_score" : {
"gte" : 3
}
}
},
"aggs" : {
"top_record_hits" : {
"top_hits" : {
"sort" : [
{
"record_score" : {
"order" : "desc"
}
}
],
"_source" : {
"includes" : [
"result_type",
"timestamp",
"record_score",
"is_interim",
"function",
"field_name",
"by_field_value",
"over_field_value",
"partition_field_value"
]
},
"size" : 3,
"script_fields" : {
"score" : {
"script" : {
"lang" : "painless",
"inline" : "Math.round(doc["record_score"].value)"
}
}
}
}
}
}
}
}
}
}
}
},

Arun_Raj · October 23, 2017, 9:02am

"condition" : {
"compare" : {
"ctx.payload.aggregations.bucket_results.doc_count" : {
"gt" : 0
}
}
},
"result" : {
"execution_time" : "2017-10-23T08:55:04.407Z",
"execution_duration" : 11,
"input" : {
"type" : "search",
"status" : "success",
"payload" : {
"_shards" : {
"total" : 15,
"failed" : 0,
"successful" : 15,
"skipped" : 0
},
"hits" : {
"hits" : [ ],
"total" : 0,
"max_score" : 0.0
},
"took" : 3,
"timed_out" : false,
"aggregations" : {
"influencer_results" : {
"doc_count" : 0,
"top_influencer_hits" : {
"hits" : {
"hits" : [ ],
"total" : 0,
"max_score" : null
}
}
},
"record_results" : {
"doc_count" : 0,
"top_record_hits" : {
"hits" : {
"hits" : [ ],
"total" : 0,
"max_score" : null
}
}
},
"bucket_results" : {
"top_bucket_hits" : {
"hits" : {
"hits" : [ ],
"total" : 0,
"max_score" : null
}
},
"doc_count" : 0
}
}
},
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
".ml-anomalies-*"
],
"types" : [ ],
"body" : {
"size" : 0,
"query" : {
"bool" : {
"filter" : [
{
"term" : {
"job_id" : "s2"
}
},
{
"range" : {
"timestamp" : {
"gte" : "now-10m"
}
}
},
{
"terms" : {
"result_type" : [
"bucket",
"record",
"influencer"
]
}
}
]
}
},
"aggs" : {
"bucket_results" : {
"filter" : {
"range" : {
"anomaly_score" : {
"gte" : 25
}
}
},
"aggs" : {
"top_bucket_hits" : {
"top_hits" : {
"sort" : [
{
"anomaly_score" : {
"order" : "desc"
}
}
],
"_source" : {
"includes" : [
"job_id",
"result_type",
"timestamp",
"anomaly_score",
"is_interim"
]
},
"size" : 1,
"script_fields" : {
"start" : {
"script" : {
"lang" : "painless",
"inline" : "new Date(doc["timestamp"].date.getMillis()-doc["bucket_span"].value * 1000 * params.padding)",
"params" : {
"padding" : 10
}
}
},
"end" : {
"script" : {
"lang" : "painless",
"inline" : "new Date(doc["timestamp"].date.getMillis()+doc["bucket_span"].value * 1000 * params.padding)",
"params" : {
"padding" : 10
}
}
},
"timestamp_epoch" : {
"script" : {
"lang" : "painless",
"inline" : "doc["timestamp"].date.getMillis()/1000"
}
},
"timestamp_iso8601" : {
"script" : {
"lang" : "painless",
"inline" : "doc["timestamp"].date"
}
},
"score" : {
"script" : {
"lang" : "painless",
"inline" : "Math.round(doc["anomaly_score"].value)"
}
}
}
}
}
}
},

Arun_Raj · October 23, 2017, 9:02am

     "influencer_results" : {
              "filter" : {
                "range" : {
                  "influencer_score" : {
                    "gte" : 3
                  }
                }
              },
              "aggs" : {
                "top_influencer_hits" : {
                  "top_hits" : {
                    "sort" : [
                      {
                        "influencer_score" : {
                          "order" : "desc"
                        }
                      }
                    ],
                    "_source" : {
                      "includes" : [
                        "result_type",
                        "timestamp",
                        "influencer_field_name",
                        "influencer_field_value",
                        "influencer_score",
                        "isInterim"
                      ]
                    },
                    "size" : 3,
                    "script_fields" : {
                      "score" : {
                        "script" : {
                          "lang" : "painless",
                          "inline" : "Math.round(doc[\"influencer_score\"].value)"
                        }
                      }
                    }
                  }
                }
              }
            },
            "record_results" : {
              "filter" : {
                "range" : {
                  "record_score" : {
                    "gte" : 3
                  }
                }
              },
              "aggs" : {
                "top_record_hits" : {
                  "top_hits" : {
                    "sort" : [
                      {
                        "record_score" : {
                          "order" : "desc"
                        }
                      }
                    ],
                    "_source" : {
                      "includes" : [
                        "result_type",
                        "timestamp",
                        "record_score",
                        "is_interim",
                        "function",
                        "field_name",
                        "by_field_value",
                        "over_field_value",
                        "partition_field_value"
                      ]
                    },
                    "size" : 3,
                    "script_fields" : {
                      "score" : {
                        "script" : {
                          "lang" : "painless",
                          "inline" : "Math.round(doc[\"record_score\"].value)"
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition" : {
    "type" : "compare",
    "status" : "success",
    "met" : false,
    "compare" : {
      "resolved_values" : {
        "ctx.payload.aggregations.bucket_results.doc_count" : 0
      }
    }
  },
  "actions" : [ ]
},
"messages" : [ ]

}
}

spinscale · October 23, 2017, 9:15am

please create a single well formatted gist out of this instead of three separate documents, this is impossible to read and follow for anyone.

See https://www.elastic.co/help

Arun_Raj · October 23, 2017, 11:41am

Sorry Alex I m getting an invalid json format even-though the content is proper as per https://jsonformatter.curiousconcept.com/

system · November 20, 2017, 11:41am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch Alert Email not working - Watcher Elasticsearch elastic-stack-alerting	12	2831	June 22, 2020
Watcher email configuration Elasticsearch elastic-stack-alerting	2	1619	December 19, 2019
Sending email while Alert Elasticsearch	8	2798	July 20, 2018
Can't send email from x-pack to gmail Elasticsearch	5	1838	December 25, 2017
Proper Configuration for SMTP Email Action in Watcher? Elasticsearch	7	8915	August 3, 2017

Action - Sending Email Alert through Gmail or Outlook

Related topics