504 timeout email watcher

Elastic Stack Elasticsearch
1

I managed to set up an email watcher on a server, with the same config on another server I getting no errors or any output in logs though the action fails. Using the following command to simulate the watcher:
POST _xpack/watcher/watch/log_errors/_execute

I get the following error after some around 40 seconds:
{ "statusCode": 504, "error": "Gateway Time-out", "message": "Client request timeout" }

This is the config for the watcher:
PUT _xpack/watcher/watch/log_errors { "trigger" : { "schedule" : { "interval" : "1m" } }, "actions" : { "send_email" : { "email" : { "to" : "my_email@bc.com", "from" : "it_test@bc.com", "subject" : "Report Test 1", "body" : "Hello from 30 box" } } } }

What could be the cause of this error which is not happening on the other server?

2

this sounds as if kibana, which is acting like a proxy, is timing out here.

  • Does the watch execute regularly?
  • Can you execute the watch via curl (you can add -m for increasing the max time to wait for the request)
  • Is there any in the watch history what might have caused this request not to return? Are you maybe tarpitted when trying to send that email or something?

A trick might be to install a local mailserver, so that watcher only delivers the mail to the local mail server and thus does not have to suffer from any timeouts/connection issues in your network or the internet when trying to send email. This should significantly speed up the watch execution as everything stays local.

--Alex

3

Yes it will likely be triggered every 5 minutes later but is currently 1 minute.

Using curl -m causes no errors to be output on the console nor is the watchers action successfully sending email and within elasticsearch logs I can see that I get a failed to execute watch log.

I'm not tarpitted as it works fine on the other server so it's strange as the settings as replicated even with mail.

4

can you share the full watch, the output of the xecute watch api from CURL and a watch history entry, please?

make sure you mask any sensitive data.

5

This is the full watch:
PUT _xpack/watcher/watch/log_errors { "trigger" : { "schedule" : { "interval" : "1m" } }, "actions" : { "send_email" : { "email" : { "to" : "my_email@bc.com", "from" : "it_test@bc.com", "subject" : "Report Test 1", "body" : "Hello from 30 box" } } } }

When I execute using curl this is the error in logs:
[WARN ][o.e.x.w.e.ExecutionService] [hostname] failed to execute watch [log_errors]

There is no new data within watch history indices, I delete the old ones previously today though I assumed new ones would created as a watch was created.

6

This is still missing the watch history output and the execute watch API output

7

The execute watch output:

{
 "statusCode": 504,
 "error": "Gateway Time-out",
 "message": "Client request timeout"
}
8

I asked for that output using curl instead of kibana as you seem to have an issue with kibana. It does not seem to correctly connect to Elasticsearch. Have you also checked that there is no network issue between the two?

9

Using curl seems to return the same old data, no new entries. There is no issue with the connection between the two.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.