While attempting to setup Watchers received a 500 error in Kibana. Log entry from Kibana:
Mar 09 08:36:58 kibana[8532]: {"type":"response","@timestamp":"2018-03-09T08:36:58Z","tags":
,"pid":8532,"method":"get","statusCode":500,"req":
{"url":"/api/watcher/watches","method":"get","headers":
{"host":"removed","connection":"keep-alive","accept":"application/json, text/plain, /","kbn-
version":"6.1.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/64.0.3282.186 Safari/537.36","referer":"/app/kibana","accept-encoding":"gzip,
deflate","accept-language":"en-
US,en;q=0.9"},"remoteAddress":"removed","userAgent":"removed","referer":"/app/kibana"},"res":
{"statusCode":500,"responseTime":22,"contentLength":9},"message":"GET /api/watcher/watches 500
22ms - 9.0B"}
Snippets of logs from Elasticsearch Log
Separate log lines
failed to set monitoring watch [l7EzHkz7SsO7BPKSzIIJgw_xpack_license_expiration]
"expiry_date": "2018-03-30T23:59:59.999Z"
[ES-Node-1] not starting watcher, upgrade API run required: .watches[false], .triggered_watches[false]
failed to set monitoring watch [l7EzHkz7SsO7BPKSzIIJgw_elasticsearch_version_mismatch]
[.watches][0] failed to execute bulk item (index) BulkShardRequest [[.watches][0]] containing [index
{[.watches][doc][l7EzHkz7SsO7BPKSzIIJgw_logstash_version_mismatch], source[n/a, actual length:
[4.1kb], max length: 2kb]}]
[.watches][0] failed to execute bulk item (index) BulkShardRequest [[.watches][0]] containing [index {[.watches][doc][l7EzHkz7SsO7BPKSzIIJgw_kibana_version_mismatch], source[n/a, actual length: [4.1kb], max length: 2kb]
Just to clarify, we are not upgrading, this is a fresh installation of Elastic stack
Version Information
./logstash --version logstash 6.1.1
./kibana --version 6.1.1
./elasticsearch --version Version: 6.1.1, Build: bd92e7f
Thank you in advance.