Illegal_state_exception Watcher status :500

alerting

(Rajasekaran Mari) #1

{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[es-master-node][10.1.1.55:9300][cluster:admin/watcher/watch/put]"
}
],
"type": "illegal_state_exception",
"reason": "not started"
},
"status": 500
}


(Alexander Reelsen) #2

Can you please take the time and write a proper description instead of just pasting a JSON response? What did you do? What do you want to do?

--Alex


(Rajasekaran Mari) #3

i tried to run this below json script
POST _watcher/watch/ccsd-snapshot/_execute

GET _watcher/watch/ccsd-snapshot
DELETE _watcher/watch/ccsd-snapshot?force
PUT _watcher/watch/ccsd-snapshot-login
{
"trigger": {
"schedule": {
"interval": "5s"
}
},
"input": {
"http": {
"request": {
"host": "localhost",
"port": 9201,
"path": "/_snapshot/temp-bck/_all"
}
}
},
"transform": {
"script": "return [total_snapshots : ctx.payload.snapshots.size()]"
},
"actions": {
"log": {
"logging": {
"text": "Found {{ctx.payload.total_snapshots}} snapshots at {{ctx.execution_time}}"
}
}
}
}


(Alexander Reelsen) #4

Hey,

have you tried the Start API, also please paste the output of the Stats API and the output of the cat shards for all watcher indices (watches, watch history, triggered watches).

--Alex


(Rajasekaran Mari) #5

Hi,

When i trying to stop the API result come true,but when i try to start the API its showing illegal_state_exception,

PUT _watcher/_stop

result showing

{
"acknowledged": true
}

PUT _watcher/_start

Result Showing

{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[es-master-node][10.1.1.55:9300][cluster:admin/watcher/service]"
}
],
"type": "null_pointer_exception",
"reason": null
},
"status": 500
}


(Alexander Reelsen) #6

Can you check the master node log file for stack traces? Can you also provide the output of the other two calls I asked for, please? Thanks!


(Rajasekaran Mari) #7

GET _watcher/stats

{
"watcher_state": "stopped",
"watch_count": 0,
"execution_thread_pool": {
"queue_size": 0,
"max_size": 0
},
"manually_stopped": true
}


(Rajasekaran Mari) #8

$tail /var/log/elasticsearch/ccsd-elk.log
at org.elasticsearch.watcher.WatcherLifeCycleService$1.beforeStop(WatcherLifeCycleService.java:57)
at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:85)
at org.elasticsearch.node.Node.stop(Node.java:326)
at org.elasticsearch.node.Node.close(Node.java:351)
at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:157)
[2016-11-11 13:03:57,283][INFO ][watcher.watch ] [es-master-node] stopped watch store
[2016-11-11 13:03:57,283][INFO ][watcher ] [es-master-node] watch service has stopped
[2016-11-11 13:03:57,303][INFO ][node ] [es-master-node] stopped
[2016-11-11 13:03:57,303][INFO ][node ] [es-master-node] closing ...
[2016-11-11 13:03:57,309][INFO ][node ] [es-master-node] closed


(Alexander Reelsen) #9

Please include more lines, not only those that were written, when you shut down elasticsearch, but also those that were written when you tried to start watcher


(Rajasekaran Mari) #10

Hi,

i tried my best ,but Watcher cannot start , i request you to please see all the previous history based on this issue. it will helpful for me. Thanks


(Alexander Reelsen) #11

The previous history does not provide enough information to help, that's the sole reason I asked for more information. If it is not provided, there is no chance to debug this issue further.


(Rajasekaran Mari) #12

Hi,

This following information its my elk log information

elk-access.log


License will expire on [Tuesday, January 31, 2017]. If you have a new license, please update it.

Otherwise, please reach out to your support contact.

Commercial plugins operate with reduced functionality on license expiration:

- marvel

- The agent will stop collecting cluster and indices metrics

- The agent will stop to automatically clean up indices older than [marvel.history.duration]

- watcher

- PUT / GET watch APIs are disabled, DELETE watch API continues to work

- Watches execute and write to the history

- The actions of the watches don't execute

[2017-01-24 22:21:36,688][WARN ][rest.suppressed ] /_watcher/_start Params: {v=}
java.lang.NullPointerException
at org.elasticsearch.watcher.execution.TriggeredWatchStore.validate(TriggeredWatchStore.java:87)
at org.elasticsearch.watcher.execution.ExecutionService.validate(ExecutionService.java:109)
at org.elasticsearch.watcher.WatcherService.validate(WatcherService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:64)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:67)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:38)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:90)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:177)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-01-24 22:22:26,832][WARN ][rest.suppressed ] /_watcher/_restart Params: {}
java.lang.NullPointerException
at org.elasticsearch.watcher.execution.TriggeredWatchStore.validate(TriggeredWatchStore.java:87)
at org.elasticsearch.watcher.execution.ExecutionService.validate(ExecutionService.java:109)
at org.elasticsearch.watcher.WatcherService.validate(WatcherService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:64)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:74)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:38)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:90)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:177)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

If want any other information i will give you.


(Rajasekaran Mari) #13

Kindly help this issue. watcher not started yet.


(Alexander Reelsen) #14

Hey,

is it possible for you share the output of GET _cluster/state? Note, this might be long, so put it in a gist or sth..

--Alex


(Alexander Reelsen) #15

In addition, can you paste the output from

GET _cat/indices/.triggered-watches

(Alexander Reelsen) #16

Quick update: Is is possible, that your .triggered_watches index is closed?

Can you open it again?

Also note, that this needs to be .triggered_watches with an underscore instead of a dash


(Rajasekaran Mari) #17

GET _cluster/state?


(Alexander Reelsen) #18

This is why I asked to put it in a gist. Please do so in order to get help.


(Rajasekaran Mari) #19

GET _cat/indices/.triggered_watches

Result



(Alexander Reelsen) #20

So seems we found your issue. You somehow closed the .triggered_watches index. You need to run

POST .triggered_watches/_open

and then you can restart watcher.