# Security Error on scheduled watches **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-alerting\ **Created:** [April 18, 2018, 12:16am UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433 "2018-04-18T00:16:27Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 18, 2018, 12:16am UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/1 "2018-04-18T00:16:28Z")
I'm able to sucessfully execute the watch with http input via enforced execution. POST \_xpack/watcher/watch/\<\>/\_execute NOTE: http input is another Elastic search. But if the same is scheduled, getting security exception on input against every scheduled run accessing input. It also provides a misleading info on input status as success. ES Version 6.2.1 "result": { "execution\_time": "2018-04-17T21:41:02.786Z", "execution\_duration": 106, "input": { "type": "http", "status": "success", "payload": { "\_headers": { "www-authenticate": [ "Basic realm="security" charset="UTF-8"" ], "content-length": [ "345" ], "content-type": [ "application/json; charset=UTF-8" ] }, "error": { "root\_cause": [ { "type": "security\_exception", "reason": "failed to authenticate user [elastic]", "header": { "WWW-Authenticate": "Basic realm="security" charset="UTF-8"" } } ], "type": "security\_exception", "reason": "failed to authenticate user [elastic]", "header": { "WWW-Authenticate": "Basic realm="security" charset="UTF-8"" } }, "\_status\_code": 401, "status": 401 } Any guidance is helpful. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 18, 2018, 3:53am UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/2 "2018-04-18T03:53:22Z")
can you share the full watch? can you share the output of the execute watch API? ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 18, 2018, 5:48am UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/3 "2018-04-18T05:48:59Z")
Hi Alex, The output is exactly as pasted in my original text. It comes under results section. The rest all are inputs. I don't have the complete text of input handy, but it is http input \_search ..ing on a indice in elastic with a simple query to check for records. The condition is if hits\>0, do the action. The es is accessed using user/password = basic authentication. Once the watch is created, and executed using xpack \_execute, it executes fine. But if the same is executed as part of schedule it fails. If it is a script error, it could be due to script issue, but the error is on security @schedule Will get the watch script though. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 18, 2018, 6:27am UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/4 "2018-04-18T06:27:31Z")
I have no doubt about the output you pasted, I just would like the whole watch plus its execution results, which is stored in that response. This eases debugging a lot. Thank you! ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 18, 2018, 6:29am UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/5 "2018-04-18T06:29:03Z")
Sure Alex. Will provide the info once I'm connected back ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 18, 2018, 7:08pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/6 "2018-04-18T19:08:45Z")
Hi Alex, Below is the watch from .watches indice { "\_index": ".watches", "\_type": "doc", "\_id": "\<\>", "\_score": 1, "\_source": { "trigger": { "schedule": { "interval": "15m" } }, "input": { "http": { "request": { "scheme": "https", "host": "\<\>", "port": 9200, "method": "get", "path": "/filebeat.log.\*/\_search", "params": {}, "headers": {}, "auth": { "basic": { "username": "elastic", "password": "::es\_encrypted::\<\<\>\>" } }, "body": """{ "size":0, "query": { "bool": { "filter": [{ "range": { "@timestamp": { "gte": "now-{{ctx.metadata.interval}}", "lte": "now" } } }] } } }""" } } }, "condition": { "compare": { "ctx.payload.hits.total": { "gt": 0 } } }, "throttle\_period\_in\_millis": 1800000, "actions": { "mailaction": { "email": { "profile": "standard", "priority": "high", "to": [ "{{ctx.metadata.alertto}}" ], "subject": "Alert - Logstash sending data to filebeat", "body": { "html": " Logstash sent {{ctx.payload.hits.total}} entries to filebeat default in last {{ctx.metadata.interval\_text}} " } } } }, "metadata": { "interval": "15m", "alertto": "\<\>", "interval\_text": "15 minutes" }, "status": { "state": { "active": true, "timestamp": "2018-04-18T00:13:13.367Z" }, "actions": { "mailaction": { "ack": { "timestamp": "2018-04-18T00:13:13.367Z", "state": "awaits\_successful\_execution" } } }, "headers": { "\_xpack\_security\_authentication": "\<\>" }, "version": -1, "last\_checked": "2018-04-18T18:28:37.613Z", "execution\_state": "execution\_not\_needed" } } } The confusing part is that the status as present in status section doesn't co-relate with watch history. The watch history indice cleary indicates the security exception, though POST \_xpack/watcher/watch/\<\>/\_execute cleanly executes the watch Regards Karthik R ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 18, 2018, 7:24pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/7 "2018-04-18T19:24:21Z")
One more question just to get a more complete picture. When you run the execute watch API, are you logged in as the same user who put the watch? ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 18, 2018, 8:14pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/8 "2018-04-18T20:14:21Z")
Hi Alex, I run the watch using my AD integrated ID (myid). When the watch is created, it is created with a user that has watcher\_admin role, which creates entry in .watches indiceThe user is native and not AD integrated. The watch by itself access another cluster using http input with appropriate user and password. The uid that we are testing with is admin in the target cluster (basic authentication) Regards Karthik R ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 18, 2018, 8:41pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/9 "2018-04-18T20:41:48Z")
I also observed that the watch fails on inputs that read from the same cluster (other than .watches indice). Should we give any alternative permissions? or execute a watch using specific user id? Regardless, the http input is explicitly specified with auth settings. Also, when I deploy the watch with a user who has admin access to the cluster where watches are deployed, it can run on schedule if the indices are read from the same cluster where watch is present. But, still fails on security as detailed above if the data is read using http input with proper security settings. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 19, 2018, 7:30am UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/10 "2018-04-19T07:30:35Z")
just to take this out of the equation: Can you store the watch as a user that is not in AD and retest? Even though the bug seems to be somewhere else, I just want to reduce the problem scope here. Also, is it possible that you changed access rights for that user some point after the watch was created (again I do not think that this is the issue, it is just about reducing problem scope). And one last thing, which may be an issue: Did you upgrade from an earlier Elasticsearch version? I see that the password is encrypted. Did you run `bin/elasticsearch-keystore add-file xpack.watcher.encryption_key /system_key` as mentiond in [https://www.elastic.co/guide/en/x-pack/6.2/encrypting-data.html](https://www.elastic.co/guide/en/x-pack/6.2/encrypting-data.html) ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 19, 2018, 7:48pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/11 "2018-04-19T19:48:08Z")
Hi Alex, We have not upgraded to any newer version, but we did respin some of of our nodes. But, we ensure that all the watches are redeployed whenever we respin any of the nodes. W.r.t creating watch, below are the scenarios tested 1. Create watch using admin user, Forced Execute - Working (Called REST API using AD user that has admin role) Scheduled Execution - Working if the data is from same cluster as watch Scheduled Execution - Not Working if the data is from another cluster accessed using http input. NOTE: http input is using basic authentication, which is set with admin user of the called cluster 2. Create watch using user with watcher\_admin role Forced Execute - Working (Called REST API using AD user has admin role) Scheduled Execution - Not Working if the data is from same cluster as watch Scheduled Execution - Not Working if the data is from another cluster accessed using http input. NOTE: http input is using basic authentication, which is set with admin user of the called cluster 3. Create watch using AD user which has admin role Forced Execute - Working (Called REST API using the same AD user that created watch) Scheduled Execution - \<\\> Scheduled Execution - Not Working if the data is from another cluster accessed using http input. NOTE: http input is using basic authentication, which is set with admin user of the called cluster Regards Karthik R ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 23, 2018, 1:39pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/12 "2018-04-23T13:39:04Z")
Hey, can you answer also the question regarding the keystore? Did you do this forall the nodes that have been respinned (as I am not sure what this means, just restarting or replacing the whole VM?). Also can you put a **full** watch history entry of a failed execution? Scrolling up I only see snippets and I would like to see one full run of possible. --Alex ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 23, 2018, 9:45pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/13 "2018-04-23T21:45:35Z")
Hi Alex, Yes, the scenarios I listed represent the same. The watch is created with a non-AD user and tested. The watches are always created with non-ad user only . I use AD user only when manually enforce watch execution using \_execute API. Also, verified that we have added the encryption file using the command exactly as mentioned in the link referenced. NOTE: Chef bash settings, that we do to set the file. Also, confirmed by looking into elasticnode checking the presence of file elasticsearch.keystore. bash "watch encryption" do user 'root' group 'elasticsearch' code \<\<-EOH echo "true" | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin xpack.watcher.encrypt\_sensitive\_data /usr/share/elasticsearch/bin/elasticsearch-keystore add-file xpack.watcher.encryption\_key #{node[:es][:config][:path]}/x-pack/system\_key EOH end NOTE: elasticsearch.keystore is not human readable file, and is expected. The strange behavior is that the same watch runs when using \_execute API explicitly, but not using the schedule. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 24, 2018, 4:33pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/14 "2018-04-24T16:33:30Z")
ok, there is one more difference. When you are using the execute watch API, then you are running the execution on the node, you are hitting it with (which means either the node you specified via curl or the node specified in your kibana configuration). When the scheduled execution happens, this happens on the same node, where the `.watches` shard is, you can see which node in the watch history entry. Can you please rerun the `elasticsearch-keystore add-file` call on those nodes holding the data and restart them? I do think that there is a divergence. The problem is, that it is not really possible to tell, if there is just stuff in the keystore or really the contents of that system key, because all you can do is `bin/elasticsearch-keystore list` and see if `xpack.watcher.encryption_key` is set in there. One thing you can ensure though, is that the md5sum/shasum of that system key is the same on all the nodes, no matter if a datanode or the node you are connecting to with kibana. ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 24, 2018, 10:18pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/15 "2018-04-24T22:18:12Z")
> [@spinscale](#): > > elasticsearch-keystore list Thanks Alex. Yes, the key is found by listing. ./elasticsearch-keystore list bootstrap.password keystore.seed xpack.watcher.encrypt\_sensitive\_data xpack.watcher.encryption\_key xpack.watcher.encrypt\_sensitive\_data is true in Yml as well, but added to keystore because of the chef setting i put earlier. It should not be a issue I believe. W.r.t Kibana - Haven't specified any specific node, rather it refers to a Load Balancer sitting on top of ES nodes. The watches sit in a cluster only with 3 nodes, and all play master/data. Will try to re-add the file though, and get back on watch history node reference. ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 24, 2018, 10:30pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/16 "2018-04-24T22:30:45Z")
Hi Alex, Also validate that the node where watch was executing (as observed from watch history) do have the xpack.watcher.encryption\_key listed. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 25, 2018, 7:12am UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/17 "2018-04-25T07:12:44Z")
Have you tried using curl to execute the watch manually on all of your nodes? Does that work as well? ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 25, 2018, 5:41pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/18 "2018-04-25T17:41:48Z")
Hi Alex, Yes. It works fine with individual es nodes, where watch shard is present. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 27, 2018, 2:00pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/19 "2018-04-27T14:00:53Z")
I was not only talking about where the watch shard is, but on **any** node in the cluster. Is this also the case? I am slowly running out of ideas, if the above works. Do you see any chance of reproduction outside of that particular system? Does the same happen if you set up a new completely independent one node cluster? ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [April 30, 2018, 9:39pm UTC](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433/20 "2018-04-30T21:39:32Z")
Hi Alex, I have a dev cluster, where the issue is same. I'm not clear on the point "but on any node in the cluster". Could you please add more details. I executed the watch from each node where a primary or replica of .watch indice being present to confirm that issue persist when executing at node directly. Sorry - I should take back my previous statement "Yes. It works fine with individual es nodes, where watch shard is present.".. Looks like I missed "doesn't". Even if I run the watch from individual nodes, I get security exception. [Next page](https://discuss.elastic.co/t/security-error-on-scheduled-watches/128433.md?page=2)