Watcher is throwing timeout_exception

[JF_LTQ (João)]

Hello,

We've been having some issues with watchers throwing timeout exceptions:

"result": {
    "execution_time": "2021-07-13T07:45:00.144Z",
    "execution_duration": 60004,
    "input": {
      "type": "search",
      "status": "failure",
      "error": {
        "root_cause": [
          {
            "type": "timeout_exception",
            "reason": "java.util.concurrent.TimeoutException: Timeout waiting for task."
          }
        ],
        "type": "timeout_exception",
        "reason": "java.util.concurrent.TimeoutException: Timeout waiting for task.",
        "caused_by": {
          "type": "timeout_exception",
          "reason": "Timeout waiting for task."
        }

This event seems to be completely random, sometimes it happens multiple times a day, sometimes it goes a few days without throwing this error.

One of the things I tried was changing these values in the elasticsearch.yml file:

xpack.http.default_connection_timeout from 120s to 240s

xpack.http.default_read_timeout from 120s to 240s

Still, it didn't work. Ideally we could maybe increase the execution_duration parameter ? If this is possible how can we do it ? I've been searching since yesterday but I haven't found anything useful.

Can anyone please assist ?

Thank you.

[spinscale (Alexander Reelsen)]

the search input is not using the http timeouts. There is a dedicated timeout parameter, see Watcher search input | Elasticsearch Guide [7.13] | Elastic

do you know why your searches take so much time? Maybe solving that would be better than increasing a timeout.

[JF_LTQ (João)]

Hey,

Thanks a lot for the reply, seems to be exactly what I need, although I'm not very experienced with elastic, could you assist me in changing the timeout parameter for all watchers?
Regarding your suggestion, I'll certainly be looking into the issue of why they are taking so long (most of them are simple queries as well), but for now I need a sort of "workaround" as we depend on these watchers for production issues.

Thank you a lot.