Illegal_state_exception Watcher status :500

Elastic Stack Elasticsearch
1

{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[es-master-node][10.1.1.55:9300][cluster:admin/watcher/watch/put]"
}
],
"type": "illegal_state_exception",
"reason": "not started"
},
"status": 500
}

2

Can you please take the time and write a proper description instead of just pasting a JSON response? What did you do? What do you want to do?

--Alex

3

i tried to run this below json script
POST _watcher/watch/ccsd-snapshot/_execute

GET _watcher/watch/ccsd-snapshot
DELETE _watcher/watch/ccsd-snapshot?force
PUT _watcher/watch/ccsd-snapshot-login
{
"trigger": {
"schedule": {
"interval": "5s"
}
},
"input": {
"http": {
"request": {
"host": "localhost",
"port": 9201,
"path": "/_snapshot/temp-bck/_all"
}
}
},
"transform": {
"script": "return [total_snapshots : ctx.payload.snapshots.size()]"
},
"actions": {
"log": {
"logging": {
"text": "Found {{ctx.payload.total_snapshots}} snapshots at {{ctx.execution_time}}"
}
}
}
}

4

Hey,

have you tried the Start API, also please paste the output of the Stats API and the output of the cat shards for all watcher indices (watches, watch history, triggered watches).

--Alex

5

Hi,

When i trying to stop the API result come true,but when i try to start the API its showing illegal_state_exception,

PUT _watcher/_stop

result showing

{
"acknowledged": true
}

PUT _watcher/_start

Result Showing

{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[es-master-node][10.1.1.55:9300][cluster:admin/watcher/service]"
}
],
"type": "null_pointer_exception",
"reason": null
},
"status": 500
}

6

Can you check the master node log file for stack traces? Can you also provide the output of the other two calls I asked for, please? Thanks!

7

GET _watcher/stats

{
"watcher_state": "stopped",
"watch_count": 0,
"execution_thread_pool": {
"queue_size": 0,
"max_size": 0
},
"manually_stopped": true
}

8

$tail /var/log/elasticsearch/ccsd-elk.log
at org.elasticsearch.watcher.WatcherLifeCycleService$1.beforeStop(WatcherLifeCycleService.java:57)
at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:85)
at org.elasticsearch.node.Node.stop(Node.java:326)
at org.elasticsearch.node.Node.close(Node.java:351)
at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:157)
[2016-11-11 13:03:57,283][INFO ][watcher.watch ] [es-master-node] stopped watch store
[2016-11-11 13:03:57,283][INFO ][watcher ] [es-master-node] watch service has stopped
[2016-11-11 13:03:57,303][INFO ][node ] [es-master-node] stopped
[2016-11-11 13:03:57,303][INFO ][node ] [es-master-node] closing ...
[2016-11-11 13:03:57,309][INFO ][node ] [es-master-node] closed

9

Please include more lines, not only those that were written, when you shut down elasticsearch, but also those that were written when you tried to start watcher

10

Hi,

i tried my best ,but Watcher cannot start , i request you to please see all the previous history based on this issue. it will helpful for me. Thanks

11

The previous history does not provide enough information to help, that's the sole reason I asked for more information. If it is not provided, there is no chance to debug this issue further.

12

Hi,

This following information its my elk log information

elk-access.log

License will expire on [Tuesday, January 31, 2017]. If you have a new license, please update it.

Otherwise, please reach out to your support contact.

Commercial plugins operate with reduced functionality on license expiration:

- marvel

- The agent will stop collecting cluster and indices metrics

- The agent will stop to automatically clean up indices older than [marvel.history.duration]

- watcher

- PUT / GET watch APIs are disabled, DELETE watch API continues to work

- Watches execute and write to the history

- The actions of the watches don't execute

[2017-01-24 22:21:36,688][WARN ][rest.suppressed ] /_watcher/_start Params: {v=}
java.lang.NullPointerException
at org.elasticsearch.watcher.execution.TriggeredWatchStore.validate(TriggeredWatchStore.java:87)
at org.elasticsearch.watcher.execution.ExecutionService.validate(ExecutionService.java:109)
at org.elasticsearch.watcher.WatcherService.validate(WatcherService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:64)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:67)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:38)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:90)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:177)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-01-24 22:22:26,832][WARN ][rest.suppressed ] /_watcher/_restart Params: {}
java.lang.NullPointerException
at org.elasticsearch.watcher.execution.TriggeredWatchStore.validate(TriggeredWatchStore.java:87)
at org.elasticsearch.watcher.execution.ExecutionService.validate(ExecutionService.java:109)
at org.elasticsearch.watcher.WatcherService.validate(WatcherService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:64)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:74)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:38)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:90)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:177)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

If want any other information i will give you.

13

Kindly help this issue. watcher not started yet.

14

Hey,

is it possible for you share the output of GET _cluster/state? Note, this might be long, so put it in a gist or sth..

--Alex

15

In addition, can you paste the output from

GET _cat/indices/.triggered-watches
16

Quick update: Is is possible, that your .triggered_watches index is closed?

Can you open it again?

Also note, that this needs to be .triggered_watches with an underscore instead of a dash

17

GET _cluster/state?

18

This is why I asked to put it in a gist. Please do so in order to get help.

19

GET _cat/indices/.triggered_watches

Result

20

So seems we found your issue. You somehow closed the .triggered_watches index. You need to run

POST .triggered_watches/_open

and then you can restart watcher.