# Watch history not written
**URL:** https://discuss.elastic.co/t/watch-history-not-written/49869
**Category:** Elasticsearch
**Tags:** elastic-stack-alerting
**Created:** [May 12, 2016, 8:43am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869 "2016-05-12T08:43:09Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 12, 2016, 8:43am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/1 "2016-05-12T08:43:09Z")
Hi!
I´m trying to add a watch with a chain input. The idea: Based on the cluster health example the watch should search in the watch\_history for the last result, check the current cluster state and trigger an action on change.
The watch works fine, but there is nothing wirtten to the watch\_history index.
The question: Why not? 🙂
Thank you!
P.S. To verify that the watch is working, I´ve used an other index with an example document.
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [May 13, 2016, 6:59am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/2 "2016-05-13T06:59:15Z")
Hey,
that sounds weird. On which Elasticsearch version are you running this?
Also could you supply us the watch, so I can reproduce it locally and debug it?
Thanks a lot!
--Alex
---
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 13, 2016, 7:39am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/3 "2016-05-13T07:39:00Z")
Thanks for your reply.
Elasticsearch, Marvel, Watcher, License in 2.3.2 (/\_cat/plugins).
```
curl -XPUT 'http://localhost:9200/_watcher/watch/cluster_health_watch' -d '{
"trigger" : {
"schedule" : { "interval" : "10s" }
},
"input" : {
"chain" : {
"inputs" : [
{
"first": {
"search": {
"request": {
"indices": [".watch_history*"],
"search_type": "query_then_fetch",
"body" : {
"query": {
"match": {
"watch_id" : "cluster_health_status"
}
},
"size": 1,
"sort": [
{
"trigger_event.triggered_time": {
"order": "desc"
}
}
]
}
}
}
}
},
{
"second": {
"http" : {
"request" : {
"host" : "localhost",
"port" : 9200,
"path" : "/_cluster/health",
"auth" : {
"basic" : {
"username" : "xxx",
"password" : "xxx"
}
}
}
}
}
}
]
}
},
"condition": {
"always": {}
},
"actions" : {
"notify-hipchat" : {
"hipchat" : {
"message" : {
"body" : "{{ctx.payload.first.hits.hits.0._source.result.input.payload.status}}",
"format" : "text",
"notify" : true
}
}
}
}
}'
```
cluster\_health\_status is the test document, based on this example: [https://www.elastic.co/guide/en/watcher/current/watch-cluster-status.html](https://www.elastic.co/guide/en/watcher/current/watch-cluster-status.html)
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [May 13, 2016, 7:45am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/4 "2016-05-13T07:45:07Z")
Hey,
I will try to run this as soon as possible (on the road right now). For some more debugging:
Run in [Execute Watch API](https://www.elastic.co/guide/en/watcher/current/api-rest.html#api-rest-execute-watch) with the `record_execution=true` parameter. You also might want to play around with the `execution_mode`
The output should help you to see if there is sth wrong and there should be history entry.
Feel free to paste any output here.
--Alex
---
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 13, 2016, 8:08am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/5 "2016-05-13T08:08:41Z")
Something new for me, thank you. Nice debug option.
```
POST _watcher/watch/cluster_health_watch/_execute
{
"record_execution" : true
}
```
The output looks fine, but no document in watch\_history. 😕
[http://pastebin.com/U7HF3aEy](http://pastebin.com/U7HF3aEy)
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [May 15, 2016, 10:56am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/7 "2016-05-15T10:56:36Z")
Hey,
I was unable to reproduce this so far. The only thing I did was havign a search history before this runs for the first time, as otherwise sorting does not work. Might that be an issue in your case?
--Alex
---
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 15, 2016, 3:10pm UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/8 "2016-05-15T15:10:42Z")
Hi,
thats right. You´ve to create the document for the search inital with an other watch or something else.
The watch works fine with the execute flag, too ☹
- Lenny
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [May 17, 2016, 6:40am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/9 "2016-05-17T06:40:53Z")
Hey,
can you share your elasticsearch config? Maybe there is a subtle difference with the hipchat account setup that is causing this...
--Alex
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [May 17, 2016, 6:42am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/10 "2016-05-17T06:42:25Z")
Hey,
also, just to make sure, the exact phrase you could grep for in your master logs would be `grep "failed to persist watch record" log-of-master.log` - I guess there is nothing either?
--Alex
---
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 17, 2016, 6:55am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/11 "2016-05-17T06:55:06Z")
Hi Alex,
you´re welcome.
```
# ---
# elasticsearch configuration
# ---
cluster.name: elasticsearch-beta
node.name: esmaster-beta01
node.master: true
node.data: false
script.inline: on
script.indexed: on
# ---
# discovery configuration
# ---
network.host: ["esmaster-beta01", "127.0.0.1"]
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["esmaster-beta01", "esmaster-beta02", "esmaster-beta03"]
# ---
# Shiled configuration
# ---
#shield.audit.enabled: true
#shield.audit.outputs: [index, logfile]
shield:
authc:
realms:
esusers:
type: esusers
order: 0
# ---
# Watcher configuration
# ---
watcher.actions.hipchat.service:
default_account: hipchat
account:
hipchat:
host:
profile: integration
auth_token:
room: Monitoring
message_defaults:
format: text
notify: true
```
---
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 17, 2016, 6:55am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/12 "2016-05-17T06:55:53Z")
```
[root@esmaster-beta01 elasticsearch]# zgrep "failed to persist watch record" elasticsearch-beta.log*
[root@esmaster-beta01 elasticsearch]#
```
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [May 25, 2016, 7:29am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/13 "2016-05-25T07:29:28Z")
Hey,
do you get any records like these in the logs?
```auto
[2016-05-25 09:28:28,590][ERROR][watcher.execution] [Angelo Unuscione] failed to update watch record [cluster_health_watch_9-2016-05-25T07:28:28.166Z]
MapperParsingException[Field name [ctx.payload.status] cannot contain '.']
```
--Alex
---
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 25, 2016, 7:43am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/14 "2016-05-25T07:43:07Z")
Hey Alex,
nope, logs looking good.
-- Lenny
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [May 25, 2016, 8:12am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/15 "2016-05-25T08:12:36Z")
Hey,
so I managed to reproduce an issue creating the watch history, when you are using chained inputs. This is the example that fails for me, however it fails with a message in the logs
```auto
PUT /_watcher/watch/cluster_health_watch
{
"trigger": {
"schedule": {
"interval": "10h"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"indices": [
".watch_history*"
],
"body": {
"query": {
"match": {
"watch_id": "cluster_health_watch"
}
},
"size": 1,
"sort": [
{
"trigger_event.triggered_time": {
"order": "desc"
}
}
]
}
}
}
}
}
]
}
},
"actions": {
"logging": {
"logging": {
"text": "Hello, I'm logging"
}
}
}
}
POST /_watcher/watch/cluster_health_watch/_execute
{
"record_execution" : true
}
```
can you try this and tell me if this creates a log entry with an exception for you?
--Alex
---
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 25, 2016, 8:35am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/16 "2016-05-25T08:35:52Z")
Hey,
something new! Thanks for this 🙂 This comes from my watch, executed with "record\_execution : true"
```
[2016-05-25 08:31:50,889][INFO][watcher.actions.logging] [esmaster-beta03] Hello, I'm logging
[2016-05-25 08:31:50,924][DEBUG][action.admin.indices.mapping.put] [esmaster-beta03] failed to put mappings on indices [[.watch_history-2016.05.25]], type [watch_record]
MapperParsingException[Field name [trigger_event.triggered_time] cannot contain '.']
```
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [May 25, 2016, 8:44am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/17 "2016-05-25T08:44:41Z")
Hey,
ah, nice, finally! 🙂
I think this is what prevents the watch history from being written. I'll work on a fix. Thx for all your patience and helpfulness!
--Alex
---
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 25, 2016, 8:49am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/18 "2016-05-25T08:49:00Z")
Hey,
thank you for your efforts!! 🙂
Lenny
---
### Author:  [@lennylinux](https://discuss.elastic.co/u/lennylinux)
#### Post date: [May 25, 2016, 9:08am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/19 "2016-05-25T09:08:15Z")
Result:
```
"watch_record": {
"watch_id": "cluster_health_watch",
"state": "executed",
"trigger_event": {
"type": "manual",
"triggered_time": "2016-05-25T09:07:53.524Z",
```
Mapping for this:
```
"triggered_time": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
```
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [May 25, 2016, 9:10am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/20 "2016-05-25T09:10:27Z")
Hey,
the problem is not the mapping of the types. The problem is that the `sort` you specified contains a dot in its fieldname. And elasticsearch tries to index this field, but this ends up being rejected...
--Alex
---
### Author:  [@bas\_vdl](https://discuss.elastic.co/u/bas_vdl)
#### Post date: [July 13, 2016, 11:09am UTC](https://discuss.elastic.co/t/watch-history-not-written/49869/21 "2016-07-13T11:09:09Z")
I'm having the same problem with ES 2.3.3. Is there a solution/workaround?
```
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"indices": [
".watch_history-*"
],
"body": {
"query": {
"match": {
"watch_id": "url"
}
},
"sort": [
{
"trigger_event.triggered_time": {
"order": "desc"
}
}
],
"size": 1
}
}
}
},
"second": {
"search": {
...
```
[Next page](https://discuss.elastic.co/t/watch-history-not-written/49869.md?page=2)