Watch history not written

lennylinux · May 12, 2016, 8:43am

Hi!

I´m trying to add a watch with a chain input. The idea: Based on the cluster health example the watch should search in the watch_history for the last result, check the current cluster state and trigger an action on change.

The watch works fine, but there is nothing wirtten to the watch_history index.

The question: Why not?

Thank you!

P.S. To verify that the watch is working, I´ve used an other index with an example document.

spinscale · May 13, 2016, 6:59am

Hey,

that sounds weird. On which Elasticsearch version are you running this?
Also could you supply us the watch, so I can reproduce it locally and debug it?

Thanks a lot!

--Alex

lennylinux · May 13, 2016, 7:39am

Thanks for your reply.

Elasticsearch, Marvel, Watcher, License in 2.3.2 (/_cat/plugins).

curl -XPUT 'http://localhost:9200/_watcher/watch/cluster_health_watch' -d '{
  "trigger" : {
    "schedule" : { "interval" : "10s" } 
  },
  "input" : {
    "chain" : {
      "inputs" : [
        {
          "first": {
             "search": {
              "request": {
                "indices": [".watch_history*"],
                "search_type": "query_then_fetch",
                "body" : {
                  "query": {
                    "match": {
                      "watch_id" : "cluster_health_status"
                    }
                  },
                  "size": 1,
                  "sort": [
                    {
                      "trigger_event.triggered_time": {
                        "order": "desc"
                      }
                    }
                  ]
                }
              }
            }
          }
        },
        {
          "second": {         
            "http" : {
              "request" : {
                "host" : "localhost",
                "port" : 9200,
                "path" : "/_cluster/health",
                "auth" : {
                  "basic" : {
                    "username" : "xxx",
                    "password" : "xxx"
                  }
                }
              }
            }
          }
        }
     ]
    }
  },
  "condition": {
    "always": {}
  },
 "actions" : {
    "notify-hipchat" : {
      "hipchat" : {
        "message" : {
          "body" : "{{ctx.payload.first.hits.hits.0._source.result.input.payload.status}}", 
          "format" : "text",
          "notify" : true
        }
      }
    }
  }
}'

cluster_health_status is the test document, based on this example: https://www.elastic.co/guide/en/watcher/current/watch-cluster-status.html

spinscale · May 13, 2016, 7:45am

Hey,

I will try to run this as soon as possible (on the road right now). For some more debugging:

Run in Execute Watch API with the record_execution=true parameter. You also might want to play around with the execution_mode

The output should help you to see if there is sth wrong and there should be history entry.

Feel free to paste any output here.

--Alex

lennylinux · May 13, 2016, 8:08am

Something new for me, thank you. Nice debug option.

POST _watcher/watch/cluster_health_watch/_execute
{
   "record_execution" : true
}

The output looks fine, but no document in watch_history.

http://pastebin.com/U7HF3aEy

spinscale · May 15, 2016, 10:56am

Hey,

I was unable to reproduce this so far. The only thing I did was havign a search history before this runs for the first time, as otherwise sorting does not work. Might that be an issue in your case?

--Alex

lennylinux · May 15, 2016, 3:10pm

Hi,

thats right. You´ve to create the document for the search inital with an other watch or something else.

The watch works fine with the execute flag, too

Lenny

spinscale · May 17, 2016, 6:40am

Hey,

can you share your elasticsearch config? Maybe there is a subtle difference with the hipchat account setup that is causing this...

--Alex

spinscale · May 17, 2016, 6:42am

Hey,

also, just to make sure, the exact phrase you could grep for in your master logs would be grep "failed to persist watch record" log-of-master.log - I guess there is nothing either?

--Alex

lennylinux · May 17, 2016, 6:55am

Hi Alex,

you´re welcome.

# ---
# elasticsearch configuration
# ---

cluster.name: elasticsearch-beta
node.name: esmaster-beta01
node.master: true
node.data: false

script.inline: on
script.indexed: on

# ---
# discovery configuration
# ---

network.host: ["esmaster-beta01", "127.0.0.1" ]
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["esmaster-beta01", "esmaster-beta02", "esmaster-beta03"]

# ---
# Shiled configuration
# ---

#shield.audit.enabled: true
#shield.audit.outputs: [index, logfile]

shield:
  authc:
    realms:
      esusers:
        type: esusers
        order: 0

# ---
# Watcher configuration
# ---

watcher.actions.hipchat.service:
  default_account: hipchat
  account:
    hipchat:
      host: <OUR_URL>
      profile: integration
      auth_token: <OUR_TOKEN>
      room: Monitoring
      message_defaults:
        format: text
        notify: true

lennylinux · May 17, 2016, 6:55am

[root@esmaster-beta01 elasticsearch]# zgrep "failed to persist watch record" elasticsearch-beta.log*
[root@esmaster-beta01 elasticsearch]#

spinscale · May 25, 2016, 7:29am

Hey,

do you get any records like these in the logs?

[2016-05-25 09:28:28,590][ERROR][watcher.execution        ] [Angelo Unuscione] failed to update watch record [cluster_health_watch_9-2016-05-25T07:28:28.166Z]
MapperParsingException[Field name [ctx.payload.status] cannot contain '.']

--Alex

lennylinux · May 25, 2016, 7:43am

Hey Alex,

nope, logs looking good.

-- Lenny

spinscale · May 25, 2016, 8:12am

Hey,

so I managed to reproduce an issue creating the watch history, when you are using chained inputs. This is the example that fails for me, however it fails with a message in the logs

PUT /_watcher/watch/cluster_health_watch
{
  "trigger": {
    "schedule": {
      "interval": "10h"
    }
  },
  "input": {
    "chain": {
      "inputs": [
        {
          "first": {
            "search": {
              "request": {
                "indices": [
                  ".watch_history*"
                ],
                "body": {
                  "query": {
                    "match": {
                      "watch_id": "cluster_health_watch"
                    }
                  },
                  "size": 1,
                  "sort": [
                    {
                      "trigger_event.triggered_time": {
                        "order": "desc"
                      }
                    }
                  ]
                }
              }
            }
          }
        }
      ]
    }
  },
  "actions": {
    "logging": {
      "logging": {
        "text": "Hello, I'm logging"
      }
    }
  }
}

POST /_watcher/watch/cluster_health_watch/_execute
{
  "record_execution" : true
}

can you try this and tell me if this creates a log entry with an exception for you?

--Alex

lennylinux · May 25, 2016, 8:35am

Hey,

something new! Thanks for this This comes from my watch, executed with "record_execution : true"

[2016-05-25 08:31:50,889][INFO ][watcher.actions.logging  ] [esmaster-beta03] Hello, I'm logging
[2016-05-25 08:31:50,924][DEBUG][action.admin.indices.mapping.put] [esmaster-beta03] failed to put mappings on indices [[.watch_history-2016.05.25]], type [watch_record]
MapperParsingException[Field name [trigger_event.triggered_time] cannot contain '.']

spinscale · May 25, 2016, 8:44am

Hey,

ah, nice, finally!
I think this is what prevents the watch history from being written. I'll work on a fix. Thx for all your patience and helpfulness!

--Alex

lennylinux · May 25, 2016, 8:49am

Hey,

thank you for your efforts!!

Lenny

lennylinux · May 25, 2016, 9:08am

Result:

  "watch_record": {
    "watch_id": "cluster_health_watch",
    "state": "executed",
    "trigger_event": {
      "type": "manual",
      "triggered_time": "2016-05-25T09:07:53.524Z",

Mapping for this:

 "triggered_time": {
   "type": "date",
   "format": "strict_date_optional_time||epoch_millis"
  },

spinscale · May 25, 2016, 9:10am

Hey,

the problem is not the mapping of the types. The problem is that the sort you specified contains a dot in its fieldname. And elasticsearch tries to index this field, but this ends up being rejected...

--Alex

bas_vdl · July 13, 2016, 11:09am

I'm having the same problem with ES 2.3.3. Is there a solution/workaround?

"input": {
    "chain": {
"inputs": [
        {
          "first": {
            "search": {
              "request": {
                "indices": [
                  ".watch_history-*"
                ],
                "body": {
                  "query": {
                    "match": {
                      "watch_id": "url"
                    }
                  },
                  "sort": [
                    {
                      "trigger_event.triggered_time": {
                        "order": "desc"
                      }
                    }
                  ],
                  "size": 1
                }
              }
            }
          },
          "second": {
            "search": {
            ...

Topic		Replies	Views
Watcher Alert mail Issue Elasticsearch elastic-stack-alerting	10	4290	July 6, 2017
Watcher chain input example, use case Elasticsearch elastic-stack-alerting	4	5450	July 6, 2017
Was the watch-history index series moved between v2 and v5 Elasticsearch elastic-stack-alerting	3	1608	March 8, 2017
Have watches created but the watches don't appear to be being triggered Elasticsearch elastic-stack-alerting	11	2377	July 6, 2017
Watcher issue (for real) Elasticsearch elastic-stack-alerting	7	881	July 6, 2017

Watch history not written

Related topics