Hi All,
Not sure if this is the correct place for this question but here goes.
I have setup watchers that generates an report and emails those out. This has worked fine untill I upgraded to ELK 6.5.4 and enabled security.
All watches fail with error:
Request Timeout after 30000ms
Digging into my kiban logs I see the following error:
{"type":"error","@timestamp":"2019-01-21T07:04:19Z","tags":[],"pid":31913,"level":"error","error":{"message":"Request Timeout after 30000ms","name":"Error","stack":"Error: Request Timeout after 30000ms\n at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15\n at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)\n at ontimeout (timers.js:498:11)\n at tryOnTimeout (timers.js:323:5)\n at Timer.listOnTimeout (timers.js:290:5)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"","query":{},"pathname":"/api/watcher/watch/execute","path":"/api/watcher/watch/execute","href":"/api/watcher/watch/execute"},"message":"Request Timeout after 30000ms"}
Where the auth field is null and then it would make sence that this requests get a timeout I think.
Here it poll if the report is generate (which isnt yet)
{"type":"response","@timestamp":"2019-01-21T07:04:47Z","tags":[],"pid":31913,"method":"get","statusCode":503,"req":{"url":"/api/reporting/jobs/download/jr5zjbes0omh3ed9ed280in6","method":"get","headers":{"accept-charset":"UTF-8","kbn-xsrf":"reporting","content-length":"0","host":"10.80.3.135:5601","connection":"Keep-Alive","user-agent":"Apache-HttpClient/4.5.2 (Java/1.8.0_162)","accept-encoding":"gzip,deflate"},"remoteAddress":"10.80.3.4","userAgent":"10.80.3.4"},"res":{"statusCode":503,"responseTime":23,"contentLength":9},"message":"GET /api/reporting/jobs/download/jr5zjbes0omh3ed9ed280in6 503 23ms - 9.0B"}
Then the error:
{"type":"error","@timestamp":"2019-01-21T07:04:19Z","tags":[],"pid":31913,"level":"error","error":{"message":"Request Timeout after 30000ms","name":"Error","stack":"Error: Request Timeout after 30000ms\n at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15\n at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)\n at ontimeout (timers.js:498:11)\n at tryOnTimeout (timers.js:323:5)\n at Timer.listOnTimeout (timers.js:290:5)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"","query":{},"pathname":"/api/watcher/watch/execute","path":"/api/watcher/watch/execute","href":"/api/watcher/watch/execute"},"message":"Request Timeout after 30000ms"}
Then an internal server error which is triggered by the Request timeout after 30000ms
{"type":"response","@timestamp":"2019-01-21T07:04:19Z","tags":[],"pid":31913,"method":"put","statusCode":500,"req":{"url":"/api/watcher/watch/execute","method":"put","headers":{"host":"logweb.tb.ukmail.iss.local","x-real-ip":"172.22.180.10","x-forwarded-for":"172.22.180.10","connection":"close","content-length":"2916","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","content-type":"application/json;charset=utf-8","kbn-version":"6.5.4"},"remoteAddress":"10.80.3.135","userAgent":"10.80.3.135"},"res":{"statusCode":500,"responseTime":30281,"contentLength":9},"message":"PUT /api/watcher/watch/execute 500 30281ms - 9.0B"}
This my kibana.yml
server.port: 5601
server.host: 10.80.3.135
server.maxPayloadBytes: 1048576
server.name: "logweb2"
elasticsearch.url: http://10.80.3.4:9200
elasticsearch.preserveHost: true
kibana.index: ".kibana-ukmail"
kibana.defaultAppId: "discover"
elasticsearch.username: elastic
elasticsearch.password: xxxxxx
xpack.monitoring.cluster_alerts.email_notifications.email_address: 'my email address'
#
elasticsearch.ssl.certificate: /etc/kibana/certs/logweb2.crt
elasticsearch.ssl.key: /etc/kibana/certs/logweb2.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" ]
elasticsearch.ssl.verificationMode: certificate
#
#elasticsearch.pingTimeout:
#elasticsearch.requestTimeout:
#elasticsearch.requestHeadersWhitelist: [ authorization ]
#elasticsearch.customHeaders: {}
#elasticsearch.shardTimeout: 0
#elasticsearch.startupTimeout: 5000
#pid.file: /var/run/kibana.pid
logging.dest: /var/log/kibana/kibana.log
#logging.silent: false
#logging.quiet: false
#logging.verbose: false
#ops.interval: 5000
xpack.security.encryptionKey: nQPdxkCTVTZ4Os0ihhOaGwGXqavih4N7ccx
#i18n.defaultLocale: "en"
xpack.reporting.enabled: true
xpack.reporting.index: ".reporting-ukmail"
xpack.reporting.encryptionKey: nQPdxkCTVTZ4Os0ihhOaGwGXqavih4N7
xpack.reporting.kibanaServer.hostname: 10.80.3.135
xpack.reporting.queue.timeout: 60000
xpack.reporting.kibanaServer.protocol: http