Watcher migration Invalid Address Error

alerting

(AbdulH) #1

I have managed to configure a one node cluster to send out email alerts, with the same configuration on a three node cluster I get the following error:

"type": "messaging_exception",
"reason": "failed to send email with subject [Report test] via account [smtp_account]",
"caused_by": {
        "type": "s_m_t_p_send_failed_exception",
        "reason": "501 5.1.7 Invalid address\n",
        "caused_by": {
          "type": "s_m_t_p_sender_failed_exception",
          "reason": "501 5.1.7 Invalid address\n"
        }
}

This is strange given that I changed the account name within each of the cluster configs to smtp_account_dev. The only meaningful difference between the two version is that the one that worked utilises elasticsearch 6.1.2 and is a one node cluster whereas the three node cluster utilise 6.1.1

This is the meaningful settings within the config:

xpack.security.enabled: false
xpack.watcher.enabled: true
xpack.notification.email.default_account: smtp_account_dev
xpack.notification.email.account.smtp_account_dev.profile: standard
xpack.notification.email.account.smtp_account_dev.smtp.host: smtp.???.??.com

Where am I going wrong?


(Alexander Reelsen) #2

can you share the full output of the Execute Watch API for this watch?

Thank you!


(AbdulH) #3
{
"_id": "vima_log_errors_ea0e2ea4-609a-4960-9ea3-dee5438b19c8-2018-01-24T09:58:44.400Z",
"watch_record": {
"watch_id": "blahblah",
"node": "oDplAK8gSQue3dcFZKxGmA",
"state": "executed",
"status": {
  "state": {
    "active": true,
    "timestamp": "2018-01-23T16:45:08.803Z"
  },
  "last_checked": "2018-01-24T09:58:44.400Z",
  "last_met_condition": "2018-01-24T09:58:44.400Z",
  "actions": {
    "send_email": {
      "ack": {
        "timestamp": "2018-01-23T16:45:08.803Z",
        "state": "awaits_successful_execution"
      },
      "last_execution": {
        "timestamp": "2018-01-24T09:58:44.400Z",
        "successful": false,
        "reason": ""
      }
    }
  },
  "execution_state": "executed",
  "version": 1026
},
"trigger_event": {
  "type": "manual",
  "triggered_time": "2018-01-24T09:58:44.399Z",
  "manual": {
    "schedule": {
      "scheduled_time": "2018-01-24T09:58:44.399Z"
    }
  }
},
"input": {
  "search": {
    "request": {
      "search_type": "query_then_fetch",
      "indices": [
        "indexes"
      ],
      "types": [],
      "body": {
        "query": {
          "range": {
            "Ticket_total": {
              "gte": 10
            }
          }
        }
      }
    }
  }
},
"condition": {
  "compare": {
    "ctx.payload.hits.total": {
      "gt": 0
    }
  }
},
"metadata": {
  "color": "yellow"
},
"result": {
  "execution_time": "2018-01-24T09:58:44.400Z",
  "execution_duration": 90,
  "input": {
    "type": "search",
    "status": "success",
    "payload": {
      "_shards": {
        "total": 5,
        "failed": 0,
        "successful": 5,
        "skipped": 0
      },
      "hits": {
        "hits": [
          ""50 RESULTS""
        ],
        "total": 50,
        "max_score": 1
      },
      "took": 3,
      "timed_out": false
    },
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "algosec_stats"
        ],
        "types": [],
        "body": {
          "query": {
            "range": {
              "Ticket_total": {
                "gte": 10
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "type": "compare",
    "status": "success",
    "met": true,
    "compare": {
      "resolved_values": {
        "ctx.payload.hits.total": 50
      }
    }
  },
  "actions": [
    {
      "id": "send_email",
      "type": "email",
      "status": "failure",
      "error": {
        "root_cause": [
          {
            "type": "messaging_exception",
            "reason": "failed to send email with subject [Report test] via account [smtp_account]"
          }
        ],
        "type": "messaging_exception",
        "reason": "failed to send email with subject [Report test] via account [smtp_account]",
        "caused_by": {
          "type": "s_m_t_p_send_failed_exception",
          "reason": "501 5.1.7 Invalid address\n",
          "caused_by": {
            "type": "s_m_t_p_sender_failed_exception",
            "reason": "501 5.1.7 Invalid address\n"
          }
        }
      }
    }
  ]
},
"messages": []
}
}

(Alexander Reelsen) #4

hey,

sorry can you include the full watch including the send_email action, that one is still missing from the above entry. Thank you so much!

--Alex


(AbdulH) #5

Apologies.

PUT _xpack/watcher/watch/vima_log_errors
{
"metadata" : {
"color" : "yellow"
},
"trigger" : {
"schedule" : {
  "interval" : "1m"
 }
},
"input" : {
"search" : {
  "request" : {
    "indices" : [ "???" ],
    "body" : {
      "query": {
         "range" : {
            "Ticket_total" : {
                 "gte" : 10
             }
         }
      }
    }
  }
}
},
"condition" : {
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
},
"actions" : {
"send_email" : {
  "email" : {
    "to" : "???@???.com",
    "from" : "???@??.com",
    "subject" : "Report test",
    "body" : "Hello World"
  }
 }
 }
}

(Alexander Reelsen) #6

Hey,

did you replace the email address by adding question marks or is that some non ASCII email address?


(AbdulH) #7

I didn't want to show my email address, it is a valid email address.


(AbdulH) #8

I found the following being logged in elasticsearch, I'm presuming my error is caused by "from:Testing" though I can't find these settings in elasticsearch.yml, where could these settings be?

[o.e.c.s.ClusterSettings  ] [hostname] updating [xpack.notification.email.] 
from [{"account":{"smtp_account":{"profile":"standard","smtp":{"host":"smtp.??.??.com"}}},
"default_account":"smtp_account"}] 
to [{"account":{"smtp_account":{"profile":"standard","smtp":
{"host":"smtp.??.??.com","user":"root@hostname.localdomain","from":"Testing"}}},
"default_account":"smtp_account"}]

(Alexander Reelsen) #9

unless you come up with concrete values it is gonna be super hard to debug this issue, as it seems that you provided an invalid email address somewhere.


(AbdulH) #10

This is the log message with full email address:

    [o.e.c.s.ClusterSettings  ] [blt] updating [xpack.notification.email.] from [{"account":{"smtp_account":{"profile":"standard","smtp":
    {"host":"smtp.intra..com"}}},"default_account":"smtp_account"}] to 
     [{"account":{"smtp_account":{"profile":"standard","smtp":
    {"host":"smtp.intra..com","user":"root@3.localdomain",
    "from":"AnalyseIT_Testing"}}},"default_account":"smtp_account"}]

This is the email address from the action:

"actions" : {
"send_email" : {
  "email" : {
    "to" : "",
    "from" : "",
    "subject" : "Report test 1",
    "body" : "Hello World"
  }
}
}

The only invalid email I can see is the "from":"AnalyseIT_Testing"}}}," though I can't find this setting anywhere within the config so I'm wondering if it persisted across upgrades and restarts.


(Alexander Reelsen) #11

you can remove the from address by simply setting it to null when you update the settings dynamically.

Do you have specified a password for SMTP auth for the root@ user? Have you tried without SMTP auth against a local SMTP server to see if that might be the culprit?


(AbdulH) #12

Should I remove from through elasticsearch.yml? I can't seem to find the setting "from":"AnalyseIT_Testing"}}}," anywhere so not where this was set.


(AbdulH) #13

It seems someone set the "from":"AnalyseIT_Testing"}}}," in cluster settings, now I need to find a way to delete these cluster settings.


(Alexander Reelsen) #14

You can use the cluster update settings API for that

https://www.elastic.co/guide/en/elasticsearch/reference/6.1/cluster-update-settings.html


(system) #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.