Watcher history indices not being created automatically

Elastic Stack Elasticsearch
1

Every watch execution throws an exception until I manually create the .watch_history- index for that day.

elasticsearch 2.0.0-beta1
watcher 2.0.0-beta1

[2015-09-14 18:50:54,727][ERROR][org.elasticsearch.watcher.execution] [Mikula Golubev] failed to update watch record [log_error_watch_56-2015-09-14T18:50:54.717Z]
[.watch_history-2015.09.14] IndexNotFoundException[no such index]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:145)
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:89)
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteSingleIndex(IndexNameExpressionResolver.java:202)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.checkBlocks(TransportReplicationAction.java:405)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:366)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:120)
at org.elasticsearch.action.index.TransportIndexAction.innerExecute(TransportIndexAction.java:140)
at org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:119)
at org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:66)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:70)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:336)
at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:354)
at org.elasticsearch.watcher.support.init.proxy.ClientProxy.index(ClientProxy.java:86)
at org.elasticsearch.watcher.history.HistoryStore.put(HistoryStore.java:87)
at org.elasticsearch.watcher.execution.ExecutionService.execute(ExecutionService.java:299)
at org.elasticsearch.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:419)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

2

Thanks for sharing this. Not sure what it is, but this shouldn't happen.

A few questions:

  1. Did you change the action.auto_create_index setting? If so can you share that setting?
  2. Can you share all your dynamic templates? (localhost:9200/_templates?pretty)
  3. Did you upgrade your cluster before you were running ES & Watcher version 2.0.0-beta1?
3

Ahh yes, it was the auto_create_index setting.
Thanks!

4

good to know and thanks for reporting this!
I think Watcher should be improved on order to make this misconfiguration clearer.

Btw you can still use the action.auto_create_index: setting by configuring the following:

action.auto_create_index: '.watches,.triggered_watches,.watch_history-*'

Only indices for watcher are then allowed to be created dynamically.