# Illegal\_state\_exception Watcher status :500 **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-alerting\ **Created:** [January 24, 2017, 7:18am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575 "2017-01-24T07:18:11Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 24, 2017, 7:18am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/1 "2017-01-24T07:18:11Z")
{ "error": { "root\_cause": [ { "type": "remote\_transport\_exception", "reason": "[es-master-node][10.1.1.55:9300][cluster:admin/watcher/watch/put]" } ], "type": "illegal\_state\_exception", "reason": "not started" }, "status": 500 } ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 24, 2017, 8:07am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/2 "2017-01-24T08:07:15Z")
Can you please take the time and write a proper description instead of just pasting a JSON response? What did you do? What do you want to do? --Alex ---
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 24, 2017, 8:13am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/3 "2017-01-24T08:13:24Z")
i tried to run this below json script POST \_watcher/watch/ccsd-snapshot/\_execute GET \_watcher/watch/ccsd-snapshot DELETE \_watcher/watch/ccsd-snapshot?force PUT \_watcher/watch/ccsd-snapshot-login { "trigger": { "schedule": { "interval": "5s" } }, "input": { "http": { "request": { "host": "localhost", "port": 9201, "path": "/\_snapshot/temp-bck/\_all" } } }, "transform": { "script": "return [total\_snapshots : ctx.payload.snapshots.size()]" }, "actions": { "log": { "logging": { "text": "Found {{ctx.payload.total\_snapshots}} snapshots at {{ctx.execution\_time}}" } } } } ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 24, 2017, 8:15am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/4 "2017-01-24T08:15:04Z")
Hey, have you tried the [Start API](https://www.elastic.co/guide/en/watcher/2.4/api-rest.html#api-rest-start), also please paste the output of the [Stats API](https://www.elastic.co/guide/en/watcher/2.4/api-rest.html#api-rest-stats) and the output of the [cat shards](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/cat-shards.html) for all watcher indices (watches, watch history, triggered watches). --Alex ---
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 24, 2017, 10:15am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/5 "2017-01-24T10:15:19Z")
Hi, When i trying to stop the API result come true,but when i try to start the API its showing illegal\_state\_exception, PUT \_watcher/\_stop **result showing** { "acknowledged": true } PUT \_watcher/\_start **Result Showing** { "error": { "root\_cause": [ { "type": "remote\_transport\_exception", "reason": "[es-master-node][10.1.1.55:9300][cluster:admin/watcher/service]" } ], "type": "null\_pointer\_exception", "reason": null }, "status": 500 } ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 24, 2017, 10:27am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/6 "2017-01-24T10:27:55Z")
Can you check the master node log file for stack traces? Can you also provide the output of the other two calls I asked for, please? Thanks! ---
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 24, 2017, 10:45am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/7 "2017-01-24T10:45:59Z")
GET \_watcher/stats { "watcher\_state": "stopped", "watch\_count": 0, "execution\_thread\_pool": { "queue\_size": 0, "max\_size": 0 }, "manually\_stopped": true } ---
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 24, 2017, 10:48am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/8 "2017-01-24T10:48:11Z")
$tail /var/log/elasticsearch/ccsd-elk.log at org.elasticsearch.watcher.WatcherLifeCycleService$1.beforeStop(WatcherLifeCycleService.java:57) at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:85) at org.elasticsearch.node.Node.stop(Node.java:326) at org.elasticsearch.node.Node.close(Node.java:351) at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:157) [2016-11-11 13:03:57,283][INFO][watcher.watch] [es-master-node] stopped watch store [2016-11-11 13:03:57,283][INFO][watcher] [es-master-node] watch service has stopped [2016-11-11 13:03:57,303][INFO][node] [es-master-node] stopped [2016-11-11 13:03:57,303][INFO][node] [es-master-node] closing ... [2016-11-11 13:03:57,309][INFO][node] [es-master-node] closed ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 24, 2017, 10:59am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/9 "2017-01-24T10:59:28Z")
Please include more lines, not only those that were written, when you shut down elasticsearch, but also those that were written when you tried to start watcher ---
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 24, 2017, 11:29am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/10 "2017-01-24T11:29:31Z")
Hi, i tried my best ,but Watcher cannot start , i request you to please see all the previous history based on this issue. it will helpful for me. Thanks ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 24, 2017, 12:16pm UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/11 "2017-01-24T12:16:22Z")
The previous history does not provide enough information to help, that's the sole reason I asked for more information. If it is not provided, there is no chance to debug this issue further. ---
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 25, 2017, 6:51am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/12 "2017-01-25T06:51:50Z")
Hi, This following information its my elk log information elk-access.log * * * # # License will expire on [Tuesday, January 31, 2017]. If you have a new license, please update it. # Otherwise, please reach out to your support contact. # # Commercial plugins operate with reduced functionality on license expiration: # - marvel # - The agent will stop collecting cluster and indices metrics # - The agent will stop to automatically clean up indices older than [marvel.history.duration] # - watcher # - PUT / GET watch APIs are disabled, DELETE watch API continues to work # - Watches execute and write to the history # - The actions of the watches don't execute [2017-01-24 22:21:36,688][WARN][rest.suppressed] /\_watcher/\_start Params: {v=} java.lang.NullPointerException at org.elasticsearch.watcher.execution.TriggeredWatchStore.validate(TriggeredWatchStore.java:87) at org.elasticsearch.watcher.execution.ExecutionService.validate(ExecutionService.java:109) at org.elasticsearch.watcher.WatcherService.validate(WatcherService.java:97) at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:97) at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:64) at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:67) at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:38) at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:90) at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:177) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [2017-01-24 22:22:26,832][WARN][rest.suppressed] /\_watcher/\_restart Params: {} java.lang.NullPointerException at org.elasticsearch.watcher.execution.TriggeredWatchStore.validate(TriggeredWatchStore.java:87) at org.elasticsearch.watcher.execution.ExecutionService.validate(ExecutionService.java:109) at org.elasticsearch.watcher.WatcherService.validate(WatcherService.java:97) at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:97) at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:64) at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:74) at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:38) at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:90) at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:177) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) If want any other information i will give you. ---
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 30, 2017, 7:13am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/13 "2017-01-30T07:13:08Z")
Kindly help this issue. watcher not started yet. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 30, 2017, 8:12am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/14 "2017-01-30T08:12:11Z")
Hey, is it possible for you share the output of `GET _cluster/state`? Note, this might be long, so put it in a gist or sth.. --Alex ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 30, 2017, 8:28am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/15 "2017-01-30T08:28:11Z")
In addition, can you paste the output from ```auto GET _cat/indices/.triggered-watches ``` ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 30, 2017, 10:14am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/16 "2017-01-30T10:14:48Z")
Quick update: Is is possible, that your `.triggered_watches` index is closed? Can you open it again? Also note, that this needs to be `.triggered_watches` with an underscore instead of a dash ---
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 30, 2017, 10:35am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/17 "2017-01-30T10:35:54Z")
GET \_cluster/state? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 30, 2017, 10:41am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/18 "2017-01-30T10:41:25Z")
This is why I asked to put it in a gist. Please do so in order to get [help](https://www.elastic.co/help). ---
**Author:** ![Rajasekaran\_Mari](https://avatars.discourse-cdn.com/v4/letter/r/53a042/32.png) [@Rajasekaran\_Mari](https://discuss.elastic.co/u/Rajasekaran_Mari)\ **Post date:** [January 30, 2017, 11:54am UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/19 "2017-01-30T11:54:36Z")
GET \_cat/indices/.triggered\_watches Result * * * ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 30, 2017, 12:15pm UTC](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575/20 "2017-01-30T12:15:10Z")
So seems we found your issue. You somehow closed the `.triggered_watches` index. You need to run ```auto POST .triggered_watches/_open ``` and then you can restart watcher. [Next page](https://discuss.elastic.co/t/illegal-state-exception-watcher-status-500/72575.md?page=2)