"watcher_state" : "starting",

Hello Team,

i have followed the startup guide to install watcher plugin using the following command.

bin/plugin -i elasticsearch/watcher/ (used the beta key as given in mail)

Plugin got installed successfully but if i see the curl -XGET 'http://username:passwd@172.0.11.104:9200/_watcher/stats?pretty'
i m getting
"watcher_state" : "starting",
"watch_count" : 0,
"execution_queue" : {
"size" : 0,
"max_size" : 0
}
}

[2015-05-22 09:54:52,775][INFO ][node ] [master1] version[1.5.1], pid[26073], build[5e38401/2015-04-09T13:41:35Z]
[2015-05-22 09:54:52,775][INFO ][node ] [master1] initializing ...
[2015-05-22 09:54:52,775][DEBUG][node ] [master1] using home [/usr/share/elasticsearch], config [/etc/elasticsearch], data [[/data]], logs [/mnt/logs], work [/tmp/elasticsearch], plugins [/usr/share/elasticsearch/plugins]
[2015-05-22 09:54:52,783][DEBUG][plugins ] [master1] lucene property is not set in plugin es-plugin.properties file. Skipping test.
[2015-05-22 09:54:52,805][DEBUG][plugins ] [master1] [/usr/share/elasticsearch/plugins/license/_site] directory does not exist.
[2015-05-22 09:54:52,808][DEBUG][plugins ] [master1] skipping [jar:file:/usr/share/elasticsearch/plugins/license/elasticsearch-license-plugin-1.0.0.jar!/es-plugin.properties]
[2015-05-22 09:54:52,808][DEBUG][plugins ] [master1] lucene property is not set in plugin es-plugin.properties file. Skipping test.
[2015-05-22 09:54:52,810][DEBUG][plugins ] [master1] skipping [jar:file:/usr/share/elasticsearch/plugins/license/elasticsearch-license-plugin-1.0.0.jar!/es-plugin.properties]
[2015-05-22 09:54:52,811][DEBUG][plugins ] [master1] skipping [jar:file:/usr/share/elasticsearch/plugins/watcher/elasticsearch-watcher-1.0.0-Beta1.jar!/es-plugin.properties]
[2015-05-22 09:54:52,811][DEBUG][plugins ] [master1] lucene property is not set in plugin es-plugin.properties file. Skipping test.
[2015-05-22 09:54:52,813][DEBUG][plugins ] [master1] [/usr/share/elasticsearch/plugins/shield/_site] directory does not exist.
[2015-05-22 09:54:52,815][DEBUG][plugins ] [master1] [/usr/share/elasticsearch/plugins/license/_site] directory does not exist.
[2015-05-22 09:54:52,816][DEBUG][plugins ] [master1] [/usr/share/elasticsearch/plugins/shield/_site] directory does not exist.
[2015-05-22 09:54:52,816][INFO ][plugins ] [master1] loaded [shield, watcher, license], sites [watcher, kopf, head]
[2015-05-22 09:54:52,844][DEBUG][common.compress.lzf ] using encoder [VanillaChunkDecoder] and decoder[{}]
[2015-05-22 09:54:52,852][DEBUG][env ] [master1] using node location [[/data/express/nodes/0]], local_node_id [0]
[2015-05-22 09:54:52,868][INFO ][watcher.trigger.schedule ] [master1] using [ticker] schedule trigger engine
[2015-05-22 09:54:53,590][INFO ][transport ] [master1] Using [org.elasticsearch.shield.transport.ShieldServerTransportService] as transport service, overridden by [shield]
[2015-05-22 09:54:53,590][INFO ][transport ] [master1] Using [org.elasticsearch.shield.transport.netty.ShieldNettyTransport] as transport, overridden by [shield]
[2015-05-22 09:54:53,590][INFO ][http ] [master1] Using [org.elasticsearch.shield.transport.netty.ShieldNettyHttpServerTransport] as http transport, overridden by [shield]
[2015-05-22 09:54:54,621][DEBUG][threadpool ] [master1] creating thread_pool [generic], type [cached], keep_alive [30s]
[2015-05-22 09:54:54,626][DEBUG][threadpool ] [master1] creating thread_pool [index], type [fixed], size [8], queue_size [200]

its alway in starting state. how to fix this .. i m using shield with elasticsearch here

Hi Amit,

From the logs it seems that node node didn't start either.

A few questions to get a better picture:

  1. Did your cluster itself actually start? Can you maybe check the cluster
    health api to check this?
  2. How many nodes are in your cluster?
  3. Do you have minimum_master_nodes configured?

Martijn

here is the output for _cluster/state

{ "cluster_name": "express","version": 11,"master_node": "RYsWWSsdT4eUdvXGfoa7IA","blocks": { },"nodes": { "KUkYNGzTRxKgvaYiBY1WIw": { "name": "master2","transport_address": "inet[/172.0.11.47:9300]","attributes": { "master": "false"}},"RYsWWSsdT4eUdvXGfoa7IA": { "name": "master1","transport_address": "inet[/172.0.11.104:9300]","attributes": { "master": "true"}}},"metadata": { "templates": { "watch_history": { "template": ".watch_history*","order": 2147483647,"settings": { "index": { "mapper": { "dynamic": "false"},"number_of_shards": "1"}},"mappings": { "watch_record": { "dynamic": "strict","dynamic_templates": [ { "disabled_payload_fields": { "mapping": { "enabled": false,"type": "object"},"match_pattern": "regex","path_match": "execution_result.((input..+)|(actions.transform..+)|(transform..+)).payload"}},{ "disabled_search_request_body_fields": { "mapping": { "enabled": false,"type": "object"},"match_pattern": "regex","path_match": "execution_result.((input)|(actions.transform.)|(transform.)).search.request.(body|template)"}}],"properties": { "message": { "type": "string"},"trigger_event": { "dynamic": true,"properties": { "schedule": { "dynamic": true,"properties": { "triggered_time": { "type": "date"},"scheduled_time": { "type": "date"}},"type": "object"}},"type": "object"},"input": { "enabled": false,"type": "object"},"execution_result": { "dynamic": true,"properties": { "transform": { "dynamic": true,"properties": { "search": { "dynamic": true,"properties": { "indices": { "index": "not_analyzed","type": "string"},"types": { "index": "not_analyzed","type": "string"}},"type": "object"}},"type": "object"},"input": { "dynamic": true,"properties": { "http": { "dynamic": true,"properties": { "request": { "dynamic": true,"properties": { "host": { "index": "not_analyzed","type": "string"},"path": { "index": "not_analyzed","type": "string"}},"type": "object"}},"type": "object"},"search": { "dynamic": true,"properties": { "request": { "dynamic": true,"properties": { "indices": { "index": "not_analyzed","type": "string"},"types": { "index": "not_analyzed","type": "string"},"search_type": { "index": "not_analyzed","type": "string"}},"type": "object"}},"type": "object"}},"type": "object"},"condition": { "dynamic": true,"properties": { "never": { "enabled": false,"type": "object"},"compare": { "enabled": false,"type": "object"},"always": { "enabled": false,"type": "object"},"script": { "enabled": false,"type": "object"}},"type": "object"},"execution_time": { "type": "date"},"actions": { "dynamic": true,"include_in_parent": true,"properties": { "index": { "dynamic": true,"properties": { "response": { "dynamic": true,"properties": { "id": { "index": "not_analyzed","type": "string"},"index": { "index": "not_analyzed","type": "string"},"type": { "index": "not_analyzed","type": "string"}},"type": "object"}},"type": "object"},"email": { "dynamic": true,"properties": { "email": { "dynamic": true,"properties": { "to": { "index": "not_analyzed","type": "string"},"id": { "index": "not_analyzed","type": "string"},"reply_to": { "index": "not_analyzed","type": "string"},"bcc": { "index": "not_analyzed","type": "string"},"from": { "index": "not_analyzed","type": "string"},"cc": { "index": "not_analyzed","type": "string"}},"type": "object"}},"type": "object"},"webhook": { "dynamic": true,"properties": { "request": { "dynamic": true,"properties": { "host": { "index": "not_analyzed","type": "string"},"path": { "index": "not_analyzed","type": "string"}},"type": "object"}},"type": "object"}},"type": "nested"}},"type": "object"},"condition": { "enabled": false,"type": "object"},"state": { "index": "not_analyzed","type": "string"},"watch_id": { "index": "not_analyzed","type": "string"},"metadata": { "dynamic": true,"type": "object"}},"_all": { "enabled": false}}}},"watches": { "template": ".watches","order": 2147483647,"settings": { "index": { "mapper": { "dynamic": "false"},"number_of_shards": "1"}},"mappings": { "watch": { "dynamic": "strict","properties": { "transform": { "dynamic": true,"enabled": false,"type": "object"},"trigger": { "dynamic": true,"enabled": false,"type": "object"},"input": { "dynamic": true,"enabled": false,"type": "object"},"status": { "dynamic": true,"enabled": false,"type": "object"},"condition": { "dynamic": true,"enabled": false,"type": "object"},"throttle_period": { "type": "string"},"actions": { "dynamic": true,"enabled": false,"type": "object"},"metadata": { "dynamic": true,"type": "object"}},"_all": { "enabled": false}}}}},"indices": { "mongodb_meta": { "state": "open","settings": { "index": { "creation_date": "1431256581195","uuid": "wn3ouTSqQnmEUEOYVY-AUw","number_of_replicas": "0","number_of_shards": "6","version": { "created": "1050199"}}},"mappings": { "mongodb_meta": { "properties": { "ns": { "type": "string"},"_ts": { "type": "long"}}}},"aliases": [ ]},"idxdispatch20150511": { "state": "open","settings": { "index": { "creation_date": "1431258484718","uuid": "4JjNKa8GTYucRtNZuLtxEw","number_of_replicas": "0","number_of_shards": "6","version": { "created": "1050199"}}},"mappings": { "string": { "properties": { "du": { "type": "string"},"itm": { "type": "string"},"col": { "type": "string"},"dwbn": { "type": "string"},"pup": { "type": "long"},"org": { "type": "string"},"bs": { "type": "string"},"vn": { "type": "string"},"kg": { "type": "string"},"cpd": { "format": "dateOptionalTime","type": "date"},"inc": { "properties": { "rmk": { "type": "string"},"dt": { "format": "dateOptionalTime","type": "date"},"wbn": { "properties": { "f": { "type": "string"},"u": { "type": "string"},"s": { "type": "string"}}},"otp": { "type": "string"},"bs": { "properties": { "u": { "type": "string"},"s": { "type": "string"}}},"user": { "type": "string"}}},"curr": { "properties": { "two": { "type": "long"},"ten": { "type": "long"},"twenty": { "type": "long"},"five_hundred": { "type": "long"},"five": { "type": "long"},"one": { "type": "long"},"hundred": { "type": "long"},"fifty": { "type": "long"},"thousand": { "type": "long"}}},"vkg": { "type": "double"},"fu": { "properties": { "username": { "type": "string"},"emp_id": { "type": "string"},"user_id": { "type": "string"},"emp_type": { "type": "string"}}},"dn": { "type": "string"},"vid": { "type": "long"},"bc": { "type": "string"},"dpd": { "format": "dateOptionalTime","type": "date"},"ds": { "index": "not_analyzed","type": "string"},"pod": { "properties": { "u": { "type": "string"},"pt": { "format": "dateOptionalTime","type": "date"},"st": { "type": "string"}}},"rmk": { "type": "string"},"dc": { "index": "not_analyzed","type": "string"},"rdt": { "format": "dateOptionalTime","type": "date"},"imei": { "type": "string"},"mage": { "type": "string"},"loc": { "type": "double"},"rou": { "properties": { "dt": { "format": "dateOptionalTime","type": "date"},"wbn": { "type": "string"},"loc": { "type": "double"},"spd": { "type": "double"},"acc": { "type": "double"}}},"cu": { "type": "string"},"fln": { "type": "string"},"u": { "type": "string"},"exp_cod": { "type": "double"},"t": { "index": "not_analyzed","type": "string"},"cn": { "index": "not_analyzed","type": "string"},"cdn": { "type": "string"},"wbn": { "type": "string"},"ac": { "type": "string"},"rs": { "properties": { "cod": { "type": "double"},"pu": { "type": "string"},"pp": { "type": "string"}}},"undelivered": { "type": "string"},"ud": { "format": "dateOptionalTime","type": "date"},"eod": { "properties": { "pd": { "type": "string"},"amt": { "type": "string"},"exp_cod": { "type": "string"},"dv": { "type": "string"},"col": { "type": "long"},"skm": { "type": "double"},"can": { "type": "long"},"ekm": { "type": "double"},"pu": { "type": "long"},"rt": { "type": "string"}}},"auth": { "type": "string"},"cd": { "format": "dateOptionalTime","type": "date"}}}},"aliases": [ "delhivery_db.dispatch"]},"kibana-int": { "state": "open","settings": { "index": { "creation_date": "1431256642348","uuid": "7IrWfFfSQcCj5uKURQJpRQ","number_of_replicas": "0","number_of_shards": "6","version": { "created": "1050199"}}},"mappings": { "dashboard": { "properties": { "title": { "type": "string"},"group": { "type": "string"},"dashboard": { "type": "string"},"user": { "type": "string"}}}},"aliases": [ ]},"idxbags20150511": { "state": "open","settings": { "index": { "creation_date": "1431257033865","uuid": "0WK72MCGRgKHKra3GQSRwA","number_of_replicas": "0","number_of_shards": "6","version": { "created": "1050199"}}},"mappings": { "string": { "properties": { "vl": { "type": "double"},"dt": { "format": "dateOptionalTime","type": "date"},"route_next": { "properties": { "connection": { "type": "long"},"origin": { "type": "string"},"departure": { "format": "dateOptionalTime","type": "date"},"arrival": { "format": "dateOptionalTime","type": "date"},"cost": { "type": "double"},"destination": { "type": "string"}}},"vh": { "type": "double"},"gmtot": { "type": "double"},"bs": { "type": "string"},"dd": { "properties": { "id": { "type": "string"},"t": { "type": "string"}}},"oc": { "index": "not_analyzed","type": "string"},"inc": { "properties": { "dt": { "format": "dateOptionalTime","type": "date"},"wbn": { "properties": { "f": { "type": "string"},"u": { "type": "string"},"s": { "type": "string"}}},"otp": { "type": "string"},"bs": { "properties": { "u": { "type": "string"},"s": { "type": "string"}}},"user": { "type": "string"}}},"fpd": { "format": "dateOptionalTime","type": "date"},"mixed": { "type": "boolean"},"force_complete": { "type": "boolean"},"vb": { "type": "double"},"rmk": { "type": "string"},"stn": { "type": "string"},"pid": { "type": "string"},"heavy": { "type": "long"},"ist": { "type": "string"},"wbns": { "type": "string"},"cs": { "properties": { "sl": { "index": "not_analyzed","type": "string"},"u": { "type": "string"},"dwbn": { "type": "string"},"ist_destination": { "type": "string"},"cn": { "type": "string"},"sr": { "type": "string"},"pbs": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "index": "not_analyzed","type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"gm": { "type": "double"},"u": { "type": "string"},"t": { "type": "string"},"cod": { "type": "double"},"cm": { "type": "boolean"},"s": { "properties": { "2": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"ist_destination": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"3": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"ist_destination": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"4": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"ist_destination": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"5": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"ist_destination": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"6": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"7": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"ist_destination": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"8": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"ist_destination": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"9": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"ist_destination": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"10": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"},"pid": { "type": "string"},"act": { "type": "string"}}},"11": { "properties": { "sl": { "type": "string"},"u": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "type": "string"}}},"sl": { "index": "not_analyzed","type": "string"},"dwbn": { "type": "string"},"sr": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"pid": { "type": "string"},"u": { "type": "string"},"ist_destination": { "type": "string"},"pbs": { "type": "string"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "index": "not_analyzed","type": "string"},"st": { "type": "string"},"act": { "type": "string"}}},"cn": { "index": "not_analyzed","type": "string"},"route": { "properties": { "connection": { "type": "long"},"origin": { "type": "string"},"departure": { "format": "dateOptionalTime","type": "date"},"arrival": { "format": "dateOptionalTime","type": "date"},"cost": { "type": "double"},"destination": { "type": "string"}}},"rs": { "type": "double"},"ud": { "format": "dateOptionalTime","type": "date"},"user": { "type": "string"},"cd": { "format": "dateOptionalTime","type": "date"},"rgn": { "type": "string"}}}},"aliases": [ "delhivery_db.bags"]},"idxpackages20150511": { "state": "open","settings": { "index": { "creation_date": "1431258406078","uuid": "WmcJyBg6Q_KUYY1P2LA-nA","number_of_replicas": "0","number_of_shards": "6","version": { "created": "1050199"}}},"mappings": { "string": { "_source": { "includes": [ "wbn","oid","bs","nm","add","prd","pin","ph","pt","cl","cn","oc","cs","ivd","pd","ldd","ed","fpd","fbd","dd","inv","zn","rs","rgn","cod","cgm","gm","cd","rcn","rpin","occ","cnc","rcc","nsl","pdd","rd","aseg"]},"properties": { "cat_notify": { "type": "boolean"},"ftd": { "format": "dateOptionalTime","type": "date"},"ns": { "type": "string"},"pdd": { "format": "dateOptionalTime","type": "date"},"etc": { "type": "string"},"tin": { "type": "string"},"dpc": { "type": "string"},"_type": { "type": "string"},"oc": { "index": "not_analyzed","type": "string"},"ncc": { "type": "string"},"od": { "format": "dateOptionalTime","type": "date"},"rdpc": { "type": "string"},"fbd": { "format": "dateOptionalTime","type": "date"},"cty": { "type": "string"},"fpd": { "format": "dateOptionalTime","type": "date"},"rst": { "type": "string"},"cgm": { "type": "double"},"st": { "type": "string"},"rph": { "type": "string"},"occ": { "index": "not_analyzed","type": "string"},"sbt": { "properties": { "tamt": { "type": "double"},"cat": { "type": "string"},"ack": { "type": "string"},"ast": { "type": "boolean"},"aid": { "type": "string"},"tval": { "type": "string"},"amt": { "type": "long"},"wbt": { "type": "boolean"},"u": { "type": "string"},"oct": { "type": "boolean"},"tax": { "type": "string"},"ort": { "type": "boolean"},"inv": { "type": "string"},"user": { "type": "string"},"cd": { "format": "dateOptionalTime","type": "date"}}},"pclh": { "type": "string"},"sc": { "type": "string"},"inv_url": { "type": "string"},"qty": { "type": "string"},"ctg": { "type": "string"},"ldd": { "format": "dateOptionalTime","type": "date"},"heavy": { "type": "long"},"sku": { "type": "string"},"oz": { "type": "string"},"si": { "type": "string"},"sadd": { "type": "string"},"cs": { "properties": { "sl": { "index": "not_analyzed","type": "string"},"dwbn": { "type": "string"},"vh": { "type": "string"},"sr": { "type": "string"},"ntd": { "format": "dateOptionalTime","type": "date"},"bucket": { "type": "string"},"sd": { "format": "dateOptionalTime","type": "date"},"loc": { "type": "double"},"pid": { "type": "string"},"asr": { "type": "string"},"add": { "type": "string"},"u": { "type": "string"},"pbs": { "type": "string"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "index": "not_analyzed","type": "string"},"st": { "type": "string"},"act": { "type": "string"},"pupid": { "type": "string"},"nsl": { "index": "not_analyzed","type": "string"}}},"radd": { "type": "string"},"dwt": { "type": "string"},"pd": { "format": "dateOptionalTime","type": "date"},"dws": { "properties": { "w": { "type": "double"},"mwt": { "type": "double"},"u": { "type": "string"},"b": { "type": "double"},"ud": { "format": "dateOptionalTime","type": "date"},"l": { "type": "double"},"h": { "type": "double"}}},"gm": { "type": "double"},"rpin": { "type": "long"},"rcn": { "type": "string"},"cn": { "index": "not_analyzed","type": "string"},"cl": { "index": "not_analyzed","type": "string"},"wbn": { "index": "not_analyzed","type": "string"},"pl": { "type": "string"},"itc": { "type": "string"},"rs": { "type": "double"},"cst": { "type": "string"},"inv": { "properties": { "dt": { "format": "dateOptionalTime","type": "date"},"rdt": { "format": "dateOptionalTime","type": "date"},"num": { "type": "string"},"rnum": { "type": "string"},"r": { "type": "string"},"zn": { "type": "string"},"rs": { "type": "string"},"tid": { "type": "string"},"ramt": { "type": "double"},"rtid": { "type": "string"},"chrgs": { "properties": { "FS": { "type": "double"},"CNC": { "type": "long"},"DL": { "type": "double"},"ST": { "type": "double"},"DTO": { "type": "long"},"COD": { "type": "long"},"RTO": { "type": "long"},"TDS": { "type": "double"}}}}},"chn": { "properties": { "dt": { "format": "dateOptionalTime","type": "date"},"n": { "type": "string"}}},"ph": { "type": "string"},"rcc": { "index": "not_analyzed","type": "string"},"nsl": { "properties": { "sl": { "type": "string"},"id": { "type": "long"},"dt": { "type": "long"},"u": { "type": "string"},"dlay": { "type": "long"},"ud": { "format": "dateOptionalTime","type": "date"},"code": { "type": "string"}}},"rgn": { "type": "string"},"cd": { "format": "dateOptionalTime","type": "date"},"cat": { "type": "string"},"rnm": { "type": "string"},"snm": { "type": "string"},"rjt": { "type": "boolean"},"date": { "properties": { "pd": { "format": "dateOptionalTime","type": "date"},"ftd": { "format": "dateOptionalTime","type": "date"},"cpd": { "format": "dateOptionalTime","type": "date"},"xpd": { "format": "dateOptionalTime","type": "date"},"lu": { "format": "dateOptionalTime","type": "date"},"mnd": { "format": "dateOptionalTime","type": "date"},"ed": { "format": "dateOptionalTime","type": "date"},"cd": { "format": "dateOptionalTime","type": "date"}}},"add": { "type": "string"},"rd": { "format": "dateOptionalTime","type": "date"},"aseg": { "properties": { "lon": { "type": "double"},"invalid_add": { "type": "boolean"},"pin": { "type": "string"},"pin_cty_mismatch": { "type": "boolean"},"loc": { "type": "string"},"aloc": { "type": "string"},"mismatch": { "type": "boolean"},"sloc": { "type": "string"},"lat": { "type": "double"}}},"ed": { "format": "dateOptionalTime","type": "date"},"dd": { "properties": { "id": { "type": "string"},"dto": { "type": "boolean"},"t": { "type": "string"},"rto": { "type": "boolean"},"dct": { "type": "long"},"srt": { "type": "boolean"},"ist": { "type": "boolean"},"vid": { "type": "long"},"rd": { "format": "dateOptionalTime","type": "date"},"fdd": { "format": "dateOptionalTime","type": "date"}}},"xpd": { "format": "dateOptionalTime","type": "date"},"cnc": { "index": "not_analyzed","type": "string"},"oid": { "index": "not_analyzed","type": "string"},"zn": { "type": "string"},"_ts": { "type": "long"},"rmk": { "type": "string"},"sid": { "type": "string"},"cnt": { "type": "string"},"lock": { "type": "boolean"},"pid": { "type": "string"},"u": { "type": "string"},"ivd": { "format": "dateOptionalTime","type": "date"},"s": { "properties": { "sl": { "type": "string"},"dwbn": { "type": "string"},"vh": { "type": "string"},"pdd": { "format": "dateOptionalTime","type": "date"},"sr": { "type": "string"},"ntd": { "format": "dateOptionalTime","type": "date"},"sd": { "format": "dateOptionalTime","type": "date"},"loc": { "type": "double"},"pid": { "type": "string"},"asr": { "type": "string"},"add": { "type": "string"},"u": { "type": "string"},"pbs": { "type": "string"},"cty": { "index": "not_analyzed","type": "string"},"prd": { "type": "string"},"ud": { "format": "dateOptionalTime","type": "date"},"ss": { "index": "not_analyzed","type": "string"},"st": { "type": "string"},"act": { "type": "string"},"pupid": { "type": "string"},"nsl": { "index": "not_analyzed","type": "string"}}},"cod": { "type": "double"},"upl": { "type": "string"},"pin": { "type": "long"},"prd": { "index": "not_analyzed","type": "string"},"pt": { "index": "not_analyzed","type": "string"},"ud": { "format": "dateOptionalTime","type": "date"},"em": { "type": "string"},"rcty": { "type": "string"},"pupid": { "type": "long"},"nm": { "type": "string"}}}},"aliases": [ "delhivery_db.packages"]},".watches": { "state": "open","settings": { "index": { "mapper": { "dynamic": "false"},"creation_date": "1432213903024","uuid": "oS3eJs_vSPGc9bJbxYsRKQ","number_of_replicas": "0","number_of_shards": "1","version": { "created": "1050199"}}},"mappings": { "watch": { "dynamic": "strict","properties": { "trigger": { "dynamic": "true","enabled": false,"type": "object"},"transform": { "dynamic": "true","enabled": false,"type": "object"},"input": { "dynamic": "true","enabled": false,"type": "object"},"status": { "dynamic": "true","enabled": false,"type": "object"},"condition": { "dynamic": "true","enabled": false,"type": "object"},"throttle_period": { "type": "string"},"actions": { "dynamic": "true","enabled": false,"type": "object"},"metadata": { "dynamic": "true","type": "object"}},"_all": { "enabled": false}}},"aliases": [ ]}}},"routing_table": { "indices": { "idxpackages20150511": { "shards": { "0": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "idxpackages20150511"}],"1": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "idxpackages20150511"}],"2": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "idxpackages20150511"}],"3": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "idxpackages20150511"}],"4": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "idxpackages20150511"}],"5": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "idxpackages20150511"}]}},".watches": { "shards": { "0": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 0,"index": ".watches"}]}},"idxdispatch20150511": { "shards": { "0": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "idxdispatch20150511"}],"1": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "idxdispatch20150511"}],"2": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "idxdispatch20150511"}],"3": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "idxdispatch20150511"}],"4": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "idxdispatch20150511"}],"5": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "idxdispatch20150511"}]}},"idxbags20150511": { "shards": { "0": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "idxbags20150511"}],"1": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "idxbags20150511"}],"2": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "idxbags20150511"}],"3": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "idxbags20150511"}],"4": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "idxbags20150511"}],"5": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "idxbags20150511"}]}},"kibana-int": { "shards": { "0": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "kibana-int"}],"1": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "kibana-int"}],"2": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "kibana-int"}],"3": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "kibana-int"}],"4": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "kibana-int"}],"5": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "kibana-int"}]}},"mongodb_meta": { "shards": { "0": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "mongodb_meta"}],"1": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "mongodb_meta"}],"2": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "mongodb_meta"}],"3": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "mongodb_meta"}],"4": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "mongodb_meta"}],"5": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "mongodb_meta"}]}}}},"routing_nodes": { "unassigned": [ ],"nodes": { "KUkYNGzTRxKgvaYiBY1WIw": [ { "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "idxpackages20150511"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "idxpackages20150511"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "idxpackages20150511"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 0,"index": ".watches"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "idxdispatch20150511"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "idxdispatch20150511"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "idxdispatch20150511"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "idxbags20150511"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "idxbags20150511"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "idxbags20150511"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "kibana-int"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "kibana-int"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "kibana-int"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 3,"index": "mongodb_meta"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 1,"index": "mongodb_meta"},{ "state": "STARTED","primary": true,"node": "KUkYNGzTRxKgvaYiBY1WIw","relocating_node": null,"shard": 5,"index": "mongodb_meta"}],"RYsWWSsdT4eUdvXGfoa7IA": [ { "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "idxpackages20150511"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "idxpackages20150511"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "idxpackages20150511"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "idxdispatch20150511"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "idxdispatch20150511"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "idxdispatch20150511"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "idxbags20150511"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "idxbags20150511"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "idxbags20150511"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "kibana-int"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "kibana-int"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "kibana-int"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 2,"index": "mongodb_meta"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 0,"index": "mongodb_meta"},{ "state": "STARTED","primary": true,"node": "RYsWWSsdT4eUdvXGfoa7IA","relocating_node": null,"shard": 4,"index": "mongodb_meta"}]}},"allocations": [ ]}

i m using one node for master see the config from master
node here

cluster.name: express
index.number_of_shards: 6
index.number_of_replicas: 0
index.mapping.ignore_malformed: true
node.name: "master1"
node.master: true
node.data: true
path.data: /data
path.logs: /mnt/logs
bootstrap.mlockall: true
network.host: 172.0.11.104
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["172.0.11.104", "172.0.11.47"]
script.disable_dynamic: false
http.jsonp.enable: false
http.basic.enabled: false
path.plugins: /usr/share/elasticsearch/plugins
shield.transport.client.auth: true
http.cors.enabled: true
http.cors.allow-origin: "/.*/"
http.cors.allow-credentials: true

any update on this?

oh, i got it . my fault, i did not install watcher in all nodes..
now it is running fine

thanks

My previous reply, somehow didn't make it to the forum...

But yes Watcher and License plugins needs to be installed on all nodes.

HMM,

thanks again