Watcher daily trigger doesn't work

Elastic Stack Elasticsearch
1

Hi

I'm using ES 2.3.5 with the compatible watcher version (2.3.5 ...).
I've been using watcher for a week now, and it works well, except one thing:
Daily trigger does not work. The watch doesn't execute.
The same watch executed just fine when i used "interval", and also when i used the Watcher REST API (_execute).

I'll explain a bit about the watch itself:
I'm using ELK Stack for log gathering and analysis, so i have daily and monthly indices, and as far as i know the best practice is to optimize every index as soon as it is not fed with new data anymore (end of day, end of month etc.).
So I have a watch that just triggers the optimize API using a webhook.
I know it's not a common watcher use case, but i don't know a better way of doing it (if you have one please let me know :slight_smile:)

Anyway, here is the watch (for a daily index optimization):

{
    "trigger": {
        "schedule": {
             "daily": {
                 "at": "03:20"
             }
        }
    },
    "transform": {
        "script": {
            "id": "daily_dateSplitScript",
            "lang": "groovy",
            "params": {}
        }
    },
    "actions": {
        "optimizeWebhook": {
            "webhook": {
                "scheme": "HTTP",
                "method": "POST",
                "host": "myHostName",                 \\ This is not the real name of course...
                "read_timeout": "1m",                 \\ In order to wait for the optimization to finish
                "port": 9200,
                "path": "/myIndexName-{{ctx.vars.newTime}}/_optimize?max_num_segments=1",
                "body": ""
        }
    }
}
}

And this is the daily_dateSplitScript indexed script:
ctx.vars.newTime = (ctx.execution_time).minusDays(1).toDate().format('YYYY.MM.dd'); return ctx.vars.newTime

As you can see, my daily indices are of the format myIndexName-YYYY.MM.dd

Again - both watch and script work together well with the interval triger. I tried more than one daily trigger (different hours).
And the watch execution info is not being indexed in the .watch_history while other watches do get indexed. It's state is "awaits_successful_execution" as if it is not triggered at all.

Can you spot my mistake?
And can you suggest any other way to optimize daily/weekly/monthly indices as they finish being fed with new data?
Thanks a lot!

2

Hey,

have you checked the master node log file, if there is any exception showing up in the logs?

--Alex

3

Hey,

another issue. Can you please show, what is returned when the watch is stored? I dont think that this watch will be stored and an error is returned, because you have to specify file instead of id, when you want to use file based scripts. Otherwise it would have been an index based scripts.

--Alex

4

Hi

Thanks Alex.
I am using an indexed script, not a file based script, so the watch uses the script as it should when it's not scheduled on a daily basis.
About the logs - you were right, there are related logs on the master. I can't print the exact log right now, but it's a parse exception. And it's quite wierd because the watch is parsed well when I use the _execute API or when I use the interval schedule. The only thing that i change between the versions is the schedule part, so it seams as if the schedule part of the watch can not be parsed when it's daily.

About your question - i dont have access to the server right now, but the return json was similar to any valid watch return json. No error or anything like it.
Same about the script.

thanks again
dan

5

So the script is not parsed for correctness when the watch is created. Please provide the exception, so further help can happen.

6

So I quickly tried the following snippet and it work. Please compare with yours, maybe there is a difference?

PUT index-2016.09.27
{
  "foo" : "bar"
}

PUT _watcher/watch/1
{
  "trigger": {
    "schedule": {
      "daily": {
        "at": "03:20"
      }
    }
  },
  "transform": {
    "script": {
      "id": "daily_dateSplitScript",
      "lang": "groovy"
    }
  },
  "actions": {
    "log": {
      "logging": {
        "text": "{{ctx.payload}}"
      }
    },
    "optimizeWebhook": {
      "webhook": {
        "scheme": "HTTP",
        "method": "POST",
        "host": "localhost",
        "read_timeout": "1m",
        "port": 9200,
        "path": "/index-{{ctx.vars.newTime}}/_optimize?max_num_segments=1"
      }
    }
  }
}

POST /_scripts/groovy/daily_dateSplitScript
{
     "script": "ctx.vars.newTime = (ctx.execution_time).minusDays(1).toDate().format('YYYY.MM.dd'); return ctx.vars.newTime"
}

PUT _watcher/watch/1/_execute

returned

{
  "_id": "1_0-2016-09-28T07:34:47.829Z",
  "watch_record": {
    "watch_id": "1",
    "state": "executed",
    "trigger_event": {
      "type": "manual",
      "triggered_time": "2016-09-28T07:34:47.829Z",
      "manual": {
        "schedule": {
          "scheduled_time": "2016-09-28T07:34:47.829Z"
        }
      }
    },
    "input": {
      "none": {}
    },
    "condition": {
      "always": {}
    },
    "messages": [],
    "result": {
      "execution_time": "2016-09-28T07:34:47.829Z",
      "execution_duration": 37,
      "input": {
        "type": "none",
        "status": "success",
        "payload": {}
      },
      "condition": {
        "type": "always",
        "status": "success",
        "met": true
      },
      "transform": {
        "type": "script",
        "status": "success",
        "payload": {
          "_value": "2016.09.27"
        }
      },
      "actions": [
        {
          "id": "log",
          "type": "logging",
          "status": "success",
          "logging": {
            "logged_text": "{_value=2016.09.27}"
          }
        },
        {
          "id": "optimizeWebhook",
          "type": "webhook",
          "status": "success",
          "webhook": {
            "request": {
              "host": "localhost",
              "port": 9200,
              "scheme": "http",
              "method": "post",
              "path": "/index-2016.09.27/_optimize?max_num_segments=1",
              "read_timeout": "1m"
            },
            "response": {
              "status": 200,
              "headers": {
                "content-length": [
                  "50"
                ],
                "content-type": [
                  "application/json; charset=UTF-8"
                ]
              },
              "body": "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
            }
          }
        }
      ]
    }
  }
}

Do you get a successful watch execution if you use the Execute Watch API

Thanks!

7

Hey

Thanks again, and sorry for the long time it took me to reply.
The watch works now the way you wrote it, but for some reason its' executions are not being listed under the .watch_history* indices.
It's the ony watch that doesn't get listed, and i can't figure out why. I don't see anything unusual in the logs.

Is it a known issue? i couldn't find anything about it...

thanks
Dan

8

Hey,

can you use the execute watch API and paste the output here?

--Alex

9

Hey

Unfortunately i can't paste it here, but:
I ran the execute API and it seems to be okay, executing correctly.
Yet - no execution recorded. Only when I add the "record_execution" : true attribute to the execute API JSON - it does record the execution.

As I stated earlier - this problem only happens with these "optimization" watches that I created.

Dan

10

If you run those watches and the optimize API kicked in, then the call might return immediately (I see you configured a timeout of one minute). Do you know if that watch actually finished when it runs once daily? You can use the watcher stats API and show the currently executed watches.

Maybe the watch has not finished

--Alex