{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[es-master-node][10.1.1.55:9300][cluster:admin/watcher/watch/put]"
}
],
"type": "illegal_state_exception",
"reason": "not started"
},
"status": 500
}
Can you please take the time and write a proper description instead of just pasting a JSON response? What did you do? What do you want to do?
--Alex
i tried to run this below json script
POST _watcher/watch/ccsd-snapshot/_execute
GET _watcher/watch/ccsd-snapshot
DELETE _watcher/watch/ccsd-snapshot?force
PUT _watcher/watch/ccsd-snapshot-login
{
"trigger": {
"schedule": {
"interval": "5s"
}
},
"input": {
"http": {
"request": {
"host": "localhost",
"port": 9201,
"path": "/_snapshot/temp-bck/_all"
}
}
},
"transform": {
"script": "return [total_snapshots : ctx.payload.snapshots.size()]"
},
"actions": {
"log": {
"logging": {
"text": "Found {{ctx.payload.total_snapshots}} snapshots at {{ctx.execution_time}}"
}
}
}
}
Hey,
have you tried the Start API, also please paste the output of the Stats API and the output of the cat shards for all watcher indices (watches, watch history, triggered watches).
--Alex
Hi,
When i trying to stop the API result come true,but when i try to start the API its showing illegal_state_exception,
PUT _watcher/_stop
result showing
{
"acknowledged": true
}
PUT _watcher/_start
Result Showing
{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[es-master-node][10.1.1.55:9300][cluster:admin/watcher/service]"
}
],
"type": "null_pointer_exception",
"reason": null
},
"status": 500
}
Can you check the master node log file for stack traces? Can you also provide the output of the other two calls I asked for, please? Thanks!
GET _watcher/stats
{
"watcher_state": "stopped",
"watch_count": 0,
"execution_thread_pool": {
"queue_size": 0,
"max_size": 0
},
"manually_stopped": true
}
$tail /var/log/elasticsearch/ccsd-elk.log
at org.elasticsearch.watcher.WatcherLifeCycleService$1.beforeStop(WatcherLifeCycleService.java:57)
at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:85)
at org.elasticsearch.node.Node.stop(Node.java:326)
at org.elasticsearch.node.Node.close(Node.java:351)
at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:157)
[2016-11-11 13:03:57,283][INFO ][watcher.watch ] [es-master-node] stopped watch store
[2016-11-11 13:03:57,283][INFO ][watcher ] [es-master-node] watch service has stopped
[2016-11-11 13:03:57,303][INFO ][node ] [es-master-node] stopped
[2016-11-11 13:03:57,303][INFO ][node ] [es-master-node] closing ...
[2016-11-11 13:03:57,309][INFO ][node ] [es-master-node] closed
Please include more lines, not only those that were written, when you shut down elasticsearch, but also those that were written when you tried to start watcher
Hi,
i tried my best ,but Watcher cannot start , i request you to please see all the previous history based on this issue. it will helpful for me. Thanks
The previous history does not provide enough information to help, that's the sole reason I asked for more information. If it is not provided, there is no chance to debug this issue further.
Hi,
This following information its my elk log information
elk-access.log
License will expire on [Tuesday, January 31, 2017]. If you have a new license, please update it.
Otherwise, please reach out to your support contact.
Commercial plugins operate with reduced functionality on license expiration:
- marvel
- The agent will stop collecting cluster and indices metrics
- The agent will stop to automatically clean up indices older than [marvel.history.duration]
- watcher
- PUT / GET watch APIs are disabled, DELETE watch API continues to work
- Watches execute and write to the history
- The actions of the watches don't execute
[2017-01-24 22:21:36,688][WARN ][rest.suppressed ] /_watcher/_start Params: {v=}
java.lang.NullPointerException
at org.elasticsearch.watcher.execution.TriggeredWatchStore.validate(TriggeredWatchStore.java:87)
at org.elasticsearch.watcher.execution.ExecutionService.validate(ExecutionService.java:109)
at org.elasticsearch.watcher.WatcherService.validate(WatcherService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:64)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:67)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:38)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:90)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:177)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2017-01-24 22:22:26,832][WARN ][rest.suppressed ] /_watcher/_restart Params: {}
java.lang.NullPointerException
at org.elasticsearch.watcher.execution.TriggeredWatchStore.validate(TriggeredWatchStore.java:87)
at org.elasticsearch.watcher.execution.ExecutionService.validate(ExecutionService.java:109)
at org.elasticsearch.watcher.WatcherService.validate(WatcherService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:97)
at org.elasticsearch.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:64)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:74)
at org.elasticsearch.watcher.transport.actions.service.TransportWatcherServiceAction.masterOperation(TransportWatcherServiceAction.java:38)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:90)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:177)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
If want any other information i will give you.
Kindly help this issue. watcher not started yet.
Hey,
is it possible for you share the output of GET _cluster/state? Note, this might be long, so put it in a gist or sth..
--Alex
In addition, can you paste the output from
GET _cat/indices/.triggered-watches
Quick update: Is is possible, that your .triggered_watches index is closed?
Can you open it again?
Also note, that this needs to be .triggered_watches with an underscore instead of a dash
GET _cluster/state?
This is why I asked to put it in a gist. Please do so in order to get help.
GET _cat/indices/.triggered_watches
Result
So seems we found your issue. You somehow closed the .triggered_watches index. You need to run
POST .triggered_watches/_open
and then you can restart watcher.