I think I totally lost the thread, I don't know where my error is right now.
I wanted to change my Elasticsearch-Kibana-WinlogBeat installation, which was working flawlessly so far, to an encrypted connection. The whole installation is running on a windows server.
With the command "elasticsearch-certutil http" I created a CSR request for our central PKI.
As response I received a .crt and a .pem file.
I included these in Elasticsearch, this is now running fine again.
But now the issue is to encrypt the connection between Kibana and Elasticsearch and Kibana and the browser. And that's where I'm just not getting anywhere at the moment.
I keep getting a "bad decrypt" error message.
Okay, thanks. I´m one step further towards the finish line
C:\_Mon\k\bin>kibana-keystore create
Created Kibana keystore in C:\_Mon\k\config\kibana.keystore
C:\_Mon\k\bin>kibana-keystore list
C:\_Mon\k\bin>kibana-keystore add server.ssl.keyPassphrase
Enter value for server.ssl.keyPassphrase: ***************************
C:\_Mon\k\bin>kibana-keystore list
server.ssl.keyPassphrase
C:\_Mon\k\bin>
Now I get a message that a first certificate cannot be verified
[2023-08-17T13:50:47.548+02:00][DEBUG][plugins.dataViewManagement] Initializing plugin
[2023-08-17T13:50:47.687+02:00][DEBUG][plugins.screenshotting.config] Running on OS: 'Win32'
[2023-08-17T13:50:47.688+02:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Win32 OS. Automatically enabling Chromium sandbox.
[2023-08-17T13:50:47.699+02:00][DEBUG][plugins.reporting] Setup complete
[2023-08-17T13:50:47.703+02:00][DEBUG][core-app] Setting up core app.
[2023-08-17T13:50:47.725+02:00][DEBUG][root] starting root
[2023-08-17T13:50:47.726+02:00][DEBUG][server] starting server
[2023-08-17T13:50:47.741+02:00][DEBUG][plugins.taskManager] status core.status.derivedStatus now set to unavailable
[2023-08-17T13:50:47.742+02:00][DEBUG][status] Recalculated core overall status
[2023-08-17T13:50:47.790+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
[2023-08-17T13:50:47.795+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
[2023-08-17T13:50:47.797+02:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate
[2023-08-17T13:50:47.804+02:00][INFO ][plugins.screenshotting.chromium] Browser executable: C:\_Mon\k\x-pack\plugins\screenshotting\chromium\chrome-win\chrome.exe
[2023-08-17T13:50:47.818+02:00][DEBUG][plugins.taskManager] status core.status.derivedStatus now set to critical
[2023-08-17T13:50:47.823+02:00][DEBUG][status] Recalculated core overall status
[2023-08-17T13:50:48.811+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
[2023-08-17T13:50:49.832+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
[2023-08-17T13:50:50.255+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
[2023-08-17T13:50:50.850+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
[2023-08-17T13:50:51.100+02:00][DEBUG][metrics.ops] memory: 294.7MB uptime: 0:00:30 load: [0.00,0.00,0.00] mean delay: 10.451 delay histogram: { 50: 10.060; 95: 11.043; 99: 12.739 }
[2023-08-17T13:50:51.870+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
[2023-08-17T13:50:52.756+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
[2023-08-17T13:50:52.889+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
[2023-08-17T13:50:53.910+02:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate
Well, Kibana doesn't trust Elasticsearch certificate for some reason. It's hard to say why elasticsearch.ssl.certificateAuthorities: config\elasticsearch-ca.pem isn't enough, maybe the ES address in the ES certificate doesn't match the address Kibana uses to connect to ES. Or maybe CA certificate is incorrect.
Try to change elasticsearch.ssl.verificationMode to none or certificate and see if it helps. By default this setting is set to full forcing Kibana to perform ES hostname verification: Configure Kibana | Kibana Guide [8.11] | Elastic
Hmm, can you check that you don't have xpack.security.http.ssl.client_authentication set to required, try to set it to none? It's fine to have this for ES node to ES node authentication ( xpack.security.transport.ssl.client_authentication), but to setup mutual PKI between Kibana and ES ( xpack.security.http.ssl.client_authentication ) you'll need to more settings for Kibana (elasticsearch.ssl.{key and certificate}).
Okay, ich checked my elastic server. Now I have another problem, maybe a connection problem which I have to solve first..
[2023-08-17T15:49:02,939][WARN ][o.e.h.AbstractHttpServerTransport] [WXTASK2P] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.20.249.89:9200, remoteAddress=/172.20.224.73:52932}io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
See logs for more details.
[2023-08-17T15:42:40,601][WARN ][o.e.h.AbstractHttpServerTransport] [WXTASK2P] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.20.249.89:9200, remoteAddress=/172.20.226.205:52239}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:286) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
... 16 more
You might even be right.
The problem is, this is a test server running multiple applications at the same time.
Elasticsearch, Kibana, WinlogBeat, Metricbeat but also a SQL server, the Windows Event Collector (WEC) and so on.
Of course, it is difficult to make a clear error analysis.
In this article Michael wrote that it is only a warning, so I continued
In this article it was also about an error with "bad_certificate".
This gave me the idea to add the certificates to the other solutions (WinlogBeat and Metricbeat) first.
Then I started elastisearch and got the bad_certificate message again.
But because it´s only a warning message, I ignored it for now.
[2023-08-18T10:15:05,004][WARN ][o.e.h.AbstractHttpServerTransport] [WXTASK2P] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.20.249.89:9200, remoteAddress=/172.20.233.189:61697}io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
at io.netty.codec@4.1.86.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.transport@4.1.86.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
See logs for more details.
Then I just started kibana - and surprise - no more errors
But I still get the message that the Kibana server is not available
But the connection is safe.
But there seems to be no error message in the kibana-system...?
Here starts my request:
[2023-08-18T10:47:29.452+02:00]
[2023-08-18T10:47:03.321+02:00][DEBUG][plugins-system.standard] Setting up plugin "upgradeAssistant"...
[2023-08-18T10:47:03.321+02:00][DEBUG][plugins.upgradeAssistant] Initializing plugin
[2023-08-18T10:47:03.327+02:00][DEBUG][plugins-system.standard] Setting up plugin "monitoring"...
[2023-08-18T10:47:03.328+02:00][DEBUG][plugins.monitoring] Initializing plugin
[2023-08-18T10:47:03.337+02:00][DEBUG][plugins-system.standard] Setting up plugin "logstash"...
[2023-08-18T10:47:03.338+02:00][DEBUG][plugins.logstash] Initializing plugin
[2023-08-18T10:47:03.339+02:00][DEBUG][plugins.logstash] Setting up Logstash plugin
[2023-08-18T10:47:03.341+02:00][DEBUG][plugins-system.standard] Setting up plugin "enterpriseSearch"...
[2023-08-18T10:47:03.342+02:00][DEBUG][plugins.enterpriseSearch] Initializing plugin
[2023-08-18T10:47:03.345+02:00][DEBUG][plugins.customIntegrations] Integration with id=ms_sql already exists.
[2023-08-18T10:47:03.414+02:00][DEBUG][plugins-system.standard] Setting up plugin "apm"...
[2023-08-18T10:47:03.415+02:00][DEBUG][plugins.apm] Initializing plugin
[2023-08-18T10:47:03.426+02:00][DEBUG][plugins.apm] Register task "apm-source-map-migration-task"
[2023-08-18T10:47:03.426+02:00][DEBUG][plugins-system.standard] Setting up plugin "visTypeGauge"...
[2023-08-18T10:47:03.427+02:00][DEBUG][plugins.visTypeGauge] Initializing plugin
[2023-08-18T10:47:03.431+02:00][DEBUG][plugins-system.standard] Setting up plugin "dataViewManagement"...
[2023-08-18T10:47:03.432+02:00][DEBUG][plugins.dataViewManagement] Initializing plugin
[2023-08-18T10:47:03.572+02:00][DEBUG][plugins.screenshotting.config] Running on OS: 'Win32'
[2023-08-18T10:47:03.572+02:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Win32 OS. Automatically enabling Chromium sandbox.
[2023-08-18T10:47:03.584+02:00][DEBUG][plugins.reporting] Setup complete
[2023-08-18T10:47:03.585+02:00][DEBUG][core-app] Setting up core app.
[2023-08-18T10:47:03.608+02:00][DEBUG][root] starting root
[2023-08-18T10:47:03.609+02:00][DEBUG][server] starting server
[2023-08-18T10:47:03.618+02:00][DEBUG][plugins.taskManager] status core.status.derivedStatus now set to unavailable
[2023-08-18T10:47:03.619+02:00][DEBUG][status] Recalculated core overall status
[2023-08-18T10:47:03.628+02:00][INFO ][plugins.screenshotting.chromium] Browser executable: C:\_Mon\k\x-pack\plugins\screenshotting\chromium\chrome-win\chrome.exe
[2023-08-18T10:47:07.010+02:00][DEBUG][metrics.ops] memory: 277.0MB uptime: 0:00:30 load: [0.00,0.00,0.00] mean delay: 15.850 delay histogram: { 50: 15.630; 95: 16.056; 99: 22.692 }
[2023-08-18T10:47:12.017+02:00][DEBUG][metrics.ops] memory: 277.1MB uptime: 0:00:35 load: [0.00,0.00,0.00] mean delay: 15.697 delay histogram: { 50: 15.630; 95: 15.770; 99: 16.695 }
[2023-08-18T10:47:17.023+02:00][DEBUG][metrics.ops] memory: 277.2MB uptime: 0:00:40 load: [0.00,0.00,0.00] mean delay: 15.741 delay histogram: { 50: 15.630; 95: 15.753; 99: 17.662 }
[2023-08-18T10:47:22.033+02:00][DEBUG][metrics.ops] memory: 277.2MB uptime: 0:00:45 load: [0.00,0.00,0.00] mean delay: 15.713 delay histogram: { 50: 15.630; 95: 15.729; 99: 17.744 }
[2023-08-18T10:47:27.046+02:00][DEBUG][metrics.ops] memory: 225.2MB uptime: 0:00:50 load: [0.00,0.00,0.00] mean delay: 16.017 delay histogram: { 50: 15.630; 95: 15.933; 99: 28.393 }
[2023-08-18T10:47:29.452+02:00][DEBUG][http.server.response] GET /login?next=%2F 200 33ms - 88.1KB
[2023-08-18T10:47:29.508+02:00][DEBUG][http.server.response] GET /node_modules/@kbn/ui-framework/dist/kui_light.min.css 304 7ms
[2023-08-18T10:47:29.509+02:00][DEBUG][http.server.response] GET /ui/legacy_light_theme.min.css 304 7ms
[2023-08-18T10:47:29.530+02:00][DEBUG][http.server.response] GET /bootstrap.js 304 3ms - 3.8KB
[2023-08-18T10:47:30.055+02:00][DEBUG][http.server.response] GET /translations/en.json 304 3ms - 29.0B
[2023-08-18T10:47:30.132+02:00][DEBUG][http.server.response] POST /api/core/capabilities 200 5ms - 46.0B
[2023-08-18T10:47:30.189+02:00][DEBUG][http.server.response] GET /internal/interactive_setup/status 503 2ms - 30.0B
[2023-08-18T10:47:30.240+02:00][DEBUG][http.server.response] GET /ui/fonts/roboto_mono/RobotoMono-Regular.ttf 304 4ms
[2023-08-18T10:47:32.056+02:00][DEBUG][metrics.ops] memory: 220.4MB uptime: 0:00:55 load: [0.00,0.00,0.00] mean delay: 16.059 delay histogram: { 50: 15.630; 95: 16.892; 99: 30.179 }
[2023-08-18T10:47:32.302+02:00][DEBUG][status] Recalculated overall status
[2023-08-18T10:47:32.709+02:00][DEBUG][status] Recalculated overall status
[2023-08-18T10:47:33.256+02:00][DEBUG][status] Recalculated overall status
[2023-08-18T10:47:37.060+02:00][DEBUG][metrics.ops] memory: 222.5MB uptime: 0:01:00 load: [0.00,0.00,0.00] mean delay: 15.691 delay histogram: { 50: 15.630; 95: 15.704; 99: 19.399 }
[2023-08-18T10:47:42.086+02:00][DEBUG][metrics.ops] memory: 222.6MB uptime: 0:01:05 load: [0.00,0.00,0.00] mean delay: 15.632 delay histogram: { 50: 15.630; 95: 15.745; 99: 15.909 }
[2023-08-18T10:47:47.092+02:00][DEBUG][metrics.ops] memory: 222.9MB uptime: 0:01:10 load: [0.00,0.00,0.00] mean delay: 15.695 delay histogram: { 50: 15.630; 95: 15.671; 99: 17.793 }
[2023-08-18T10:47:52.097+02:00][DEBUG][metrics.ops] memory: 222.9MB uptime: 0:01:15 load: [0.00,0.00,0.00] mean delay: 15.639 delay histogram: { 50: 15.630; 95: 15.671; 99: 15.753 }
It seems to be that everything is okay, but it doesn´t work
First:
The problem is solved, the SSL connection works now.
I think I have to summarize all this in a separate thread in the next few days..
I´ve received these messages:
[2023-08-18T13:34:48.329+02:00][DEBUG][elasticsearch.query.data] 401 - 615.0B
GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip [security_exception]: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
[2023-08-18T13:34:48.938+02:00][DEBUG][elasticsearch.query.data] 401 - 459.0B
GET / [security_exception]: missing authentication credentials for REST request [/]
I read in an article that you have to enter the username and password in the kibana.yml:
After trying this in plain text with the Elastic user, I received an error message:
FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html
With this message and this web address I created the token:
In the next step I then tried to test this with Curl
C:\Users\273872a\Documents\curl\bin>curl -H "Authorization: Bearer AAEAA...lBX09oZw" https://wxtask2p.idm.lan.local:9200/_cluster/health
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
This failed because I didn´t specify the certificate.
Second try:
After that I restarted elasticsearch (although it was probably not necessary) and then kibana.
And suddenly everything works. The connection is secured and logging in also works without problems.
However, the kibana system shows me some strange and long messages.
QUESTION:
Is this normal?
[2023-08-18T14:21:20.099+02:00][DEBUG][elasticsearch.query.data] 200 - 227.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n if (params.claimableTaskTypes.contains(ctx._source.task.taskType)) {\n if (ctx._source.task.schedule != null || ctx._source.task.attempts < params.taskMaxAttempts[ctx._source.task.taskType]) {\n if(ctx._source.task.retryAt != null && ZonedDateTime.parse(ctx._source.task.retryAt).toInstant().toEpochMilli() < params.now) {\n ctx._source.task.scheduledAt=ctx._source.task.retryAt;\n } else {\n ctx._source.task.scheduledAt=ctx._source.task.runAt;\n }\n ctx._source.task.status = \"claiming\"; ctx._source.task.ownerId=params.fieldUpdates.ownerId; ctx._source.task.retryAt=params.fieldUpdates.retryAt;\n } else {\n ctx._source.task.status = \"failed\";\n }\n } else if (params.unusedTaskTypes.contains(ctx._source.task.taskType)) {\n ctx._source.task.status = \"unrecognized\";\n } else {\n ctx.op = \"noop\";\n }","lang":"painless","params":{"now":1692361280094,"fieldUpdates":{"ownerId":"kibana:3cc50870-5b0a-4625-a563-055064a00ef7","retryAt":"2023-08-18T12:21:50.073Z"},"claimableTaskTypes":["session_cleanup","actions_telemetry","cleanup_failed_action_executions","alerting_telemetry","alerts_invalidate_api_keys","alerting_health_check","reports:monitor","alerting:transform_health","actions:.email","actions:.index","actions:.pagerduty","actions:.swimlane","actions:.server-log","actions:.slack","actions:.webhook","actions:.cases-webhook","actions:.xmatters","actions:.servicenow","actions:.servicenow-sir","actions:.servicenow-itom","actions:.jira","actions:.resilient","actions:.teams","actions:.torq","actions:.opsgenie","actions:.tines","alerting:.index-threshold","alerting:.geo-containment","alerting:.es-query","dashboard_telemetry","cases-telemetry-task","Fleet-Usage-Sender","Fleet-Usage-Logger","fleet:reassign_action:retry","fleet:unenroll_action:retry","fleet:upgrade_action:retry","fleet:update_agent_tags:retry","fleet:request_diagnostics:retry","fleet:check-deleted-files-task","osquery:telemetry-packs","osquery:telemetry-saved-queries","osquery:telemetry-configs","cloud_security_posture-stats_task","ML:saved-objects-sync","alerting:xpack.ml.anomaly_detection_alert","alerting:xpack.ml.anomaly_detection_jobs_health","UPTIME:SyntheticsService:Sync-Saved-Monitor-Objects","alerting:xpack.uptime.alerts.monitorStatus","alerting:xpack.uptime.alerts.tlsCertificate","alerting:xpack.uptime.alerts.durationAnomaly","alerting:xpack.uptime.alerts.tls","alerting:xpack.synthetics.alerts.monitorStatus","alerting:siem.eqlRule","alerting:siem.savedQueryRule","alerting:siem.indicatorRule","alerting:siem.mlRule","alerting:siem.queryRule","alerting:siem.thresholdRule","alerting:siem.newTermsRule","alerting:siem.notifications","endpoint:user-artifact-packager","security:endpoint-diagnostics","security:endpoint-meta-telemetry","security:telemetry-lists","security:telemetry-detection-rules","security:telemetry-prebuilt-rule-alerts","security:telemetry-timelines","security:telemetry-configuration","security:telemetry-filterlist-artifact","endpoint:metadata-check-transforms-task","alerting:metrics.alert.anomaly","alerting:logs.alert.document.count","alerting:metrics.alert.inventory.threshold","alerting:metrics.alert.threshold","alerting:monitoring_alert_cluster_health","alerting:monitoring_alert_license_expiration","alerting:monitoring_alert_cpu_usage","alerting:monitoring_alert_missing_monitoring_data","alerting:monitoring_alert_disk_usage","alerting:monitoring_alert_thread_pool_search_rejections","alerting:monitoring_alert_thread_pool_write_rejections","alerting:monitoring_alert_jvm_memory_usage","alerting:monitoring_alert_nodes_changed","alerting:monitoring_alert_logstash_version_mismatch","alerting:monitoring_alert_kibana_version_mismatch","alerting:monitoring_alert_elasticsearch_version_mismatch","alerting:monitoring_ccr_read_exceptions","alerting:monitoring_shard_size","apm-telemetry-task","alerting:apm.transaction_duration","alerting:apm.anomaly","alerting:apm.error_rate","alerting:apm.transaction_error_rate"],"skippedTaskTypes":["report:execute","apm-source-map-migration-task"],"unusedTaskTypes":["sampleTaskRemovedType","alerting:siem.signals","search_sessions_monitor","search_sessions_cleanup","search_sessions_expire"],"taskMaxAttempts":{"session_cleanup":3,"actions_telemetry":3,"cleanup_failed_action_executions":3,"alerting_telemetry":3,"alerts_invalidate_api_keys":3,"alerting_health_check":3,"reports:monitor":1,"alerting:transform_health":3,"actions:.email":3,"actions:.index":3,"actions:.pagerduty":3,"actions:.swimlane":3,"actions:.server-log":3,"actions:.slack":3,"actions:.webhook":3,"actions:.cases-webhook":3,"actions:.xmatters":3,"actions:.servicenow":3,"actions:.servicenow-sir":3,"actions:.servicenow-itom":3,"actions:.jira":3,"actions:.resilient":3,"actions:.teams":3,"actions:.torq":3,"actions:.opsgenie":3,"actions:.tines":3,"alerting:.index-threshold":3,"alerting:.geo-containment":3,"alerting:.es-query":3,"dashboard_telemetry":3,"cases-telemetry-task":3,"Fleet-Usage-Sender":1,"Fleet-Usage-Logger":1,"fleet:reassign_action:retry":1,"fleet:unenroll_action:retry":1,"fleet:upgrade_action:retry":1,"fleet:update_agent_tags:retry":1,"fleet:request_diagnostics:retry":1,"fleet:check-deleted-files-task":3,"osquery:telemetry-packs":3,"osquery:telemetry-saved-queries":3,"osquery:telemetry-configs":3,"cloud_security_posture-stats_task":3,"ML:saved-objects-sync":3,"alerting:xpack.ml.anomaly_detection_alert":3,"alerting:xpack.ml.anomaly_detection_jobs_health":3,"UPTIME:SyntheticsService:Sync-Saved-Monitor-Objects":3,"alerting:xpack.uptime.alerts.monitorStatus":3,"alerting:xpack.uptime.alerts.tlsCertificate":3,"alerting:xpack.uptime.alerts.durationAnomaly":3,"alerting:xpack.uptime.alerts.tls":3,"alerting:xpack.synthetics.alerts.monitorStatus":3,"alerting:siem.eqlRule":3,"alerting:siem.savedQueryRule":3,"alerting:siem.indicatorRule":3,"alerting:siem.mlRule":3,"alerting:siem.queryRule":3,"alerting:siem.thresholdRule":3,"alerting:siem.newTermsRule":3,"alerting:siem.notifications":3,"endpoint:user-artifact-packager":3,"security:endpoint-diagnostics":3,"security:endpoint-meta-telemetry":3,"security:telemetry-lists":3,"security:telemetry-detection-rules":3,"security:telemetry-prebuilt-rule-alerts":3,"security:telemetry-timelines":3,"security:telemetry-configuration":3,"security:telemetry-filterlist-artifact":3,"endpoint:metadata-check-transforms-task":3,"alerting:metrics.alert.anomaly":3,"alerting:logs.alert.document.count":3,"alerting:metrics.alert.inventory.threshold":3,"alerting:metrics.alert.threshold":3,"alerting:monitoring_alert_cluster_health":3,"alerting:monitoring_alert_license_expiration":3,"alerting:monitoring_alert_cpu_usage":3,"alerting:monitoring_alert_missing_monitoring_data":3,"alerting:monitoring_alert_disk_usage":3,"alerting:monitoring_alert_thread_pool_search_rejections":3,"alerting:monitoring_alert_thread_pool_write_rejections":3,"alerting:monitoring_alert_jvm_memory_usage":3,"alerting:monitoring_alert_nodes_changed":3,"alerting:monitoring_alert_logstash_version_mismatch":3,"alerting:monitoring_alert_kibana_version_mismatch":3,"alerting:monitoring_alert_elasticsearch_version_mismatch":3,"alerting:monitoring_ccr_read_exceptions":3,"alerting:monitoring_shard_size":3,"apm-telemetry-task":3,"alerting:apm.transaction_duration":3,"alerting:apm.anomaly":3,"alerting:apm.error_rate":3,"alerting:apm.transaction_error_rate":3}}},"sort":[{"_script":{"type":"number","order":"asc","script":{"lang":"painless","source":"\nif (doc['task.retryAt'].size()!=0) {\n return doc['task.retryAt'].value.toInstant().toEpochMilli();\n}\nif (doc['task.runAt'].size()!=0) {\n return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n "}}}],"max_docs":10,"conflicts":"proceed"}
[2023-08-18T14:21:20.591+02:00][DEBUG][elasticsearch.query.data] 200 - 125.0B
GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip
[2023-08-18T14:21:20.667+02:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: Error connecting to package registry: request to https://epr.elastic.co/search?package=synthetics&prerelease=true&kibana.version=8.7.0 failed, reason: connect ETIMEDOUT 34.120.127.130:443
[2023-08-18T14:21:20.671+02:00][DEBUG][elasticsearch.query.data] 200 - 10.3KB
GET /.kibana_8.7.0/_doc/epm-packages%3Asynthetics
[2023-08-18T14:21:20.671+02:00][INFO ][plugins.synthetics] Installed synthetics index templates
[2023-08-18T14:21:22.905+02:00][DEBUG][elasticsearch.query.monitoring] 200 - 1.3KB
GET /_xpack
[2023-08-18T14:21:22.937+02:00][DEBUG][elasticsearch.query.data] 200 - 399.0B
GET /.kibana_8.7.0/_doc/telemetry%3Atelemetry
[2023-08-18T14:21:23.016+02:00][DEBUG][metrics.ops] memory: 275.0MB uptime: 0:02:02 load: [0.00,0.00,0.00] mean delay: 15.725 delay histogram: { 50: 15.630; 95: 16.163; 99: 24.134 }
[2023-08-18T14:21:23.094+02:00][DEBUG][elasticsearch.query.data] 200 - 125.0B
GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip
[2023-08-18T14:21:23.099+02:00][DEBUG][elasticsearch.query.data] 200 - 227.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n if (params.claimableTaskTypes.contains(ctx._source.task.taskType)) {\n if (ctx._source.task.schedule != null || ctx._source.task.attempts < params.taskMaxAttempts[ctx._source.task.taskType]) {\n if(ctx._source.task.retryAt != null && ZonedDateTime.parse(ctx._source.task.retryAt).toInstant().toEpochMilli() < params.now) {\n ctx._source.task.scheduledAt=ctx._source.task.retryAt;\n } else {\n ctx._source.task.scheduledAt=ctx._source.task.runAt;\n }\n ctx._source.task.status = \"claiming\"; ctx._source.task.ownerId=params.fieldUpdates.ownerId; ctx._source.task.retryAt=params.fieldUpdates.retryAt;\n } else {\n ctx._source.task.status = \"failed\";\n }\n } else if (params.unusedTaskTypes.contains(ctx._source.task.taskType)) {\n ctx._source.task.status = \"unrecognized\";\n } else {\n ctx.op = \"noop\";\n }","lang":"painless","params":{"now":1692361283091,"fieldUpdates":{"ownerId":"kibana:3cc50870-5b0a-4625-a563-055064a00ef7","retryAt":"2023-08-18T12:21:53.090Z"},"claimableTaskTypes":["apm-source-map-migration-task"],"skippedTaskTypes":["session_cleanup","actions_telemetry","cleanup_failed_action_executions","alerting_telemetry","alerts_invalidate_api_keys","alerting_health_check","report:execute","reports:monitor","alerting:transform_health","actions:.email","actions:.index","actions:.pagerduty","actions:.swimlane","actions:.server-log","actions:.slack","actions:.webhook","actions:.cases-webhook","actions:.xmatters","actions:.servicenow","actions:.servicenow-sir","actions:.servicenow-itom","actions:.jira","actions:.resilient","actions:.teams","actions:.torq","actions:.opsgenie","actions:.tines","alerting:.index-threshold","alerting:.geo-containment","alerting:.es-query","dashboard_telemetry","cases-telemetry-task","Fleet-Usage-Sender","Fleet-Usage-Logger","fleet:reassign_action:retry","fleet:unenroll_action:retry","fleet:upgrade_action:retry","fleet:update_agent_tags:retry","fleet:request_diagnostics:retry","fleet:check-deleted-files-task","osquery:telemetry-packs","osquery:telemetry-saved-queries","osquery:telemetry-configs","cloud_security_posture-stats_task","ML:saved-objects-sync","alerting:xpack.ml.anomaly_detection_alert","alerting:xpack.ml.anomaly_detection_jobs_health","UPTIME:SyntheticsService:Sync-Saved-Monitor-Objects","alerting:xpack.uptime.alerts.monitorStatus","alerting:xpack.uptime.alerts.tlsCertificate","alerting:xpack.uptime.alerts.durationAnomaly","alerting:xpack.uptime.alerts.tls","alerting:xpack.synthetics.alerts.monitorStatus","alerting:siem.eqlRule","alerting:siem.savedQueryRule","alerting:siem.indicatorRule","alerting:siem.mlRule","alerting:siem.queryRule","alerting:siem.thresholdRule","alerting:siem.newTermsRule","alerting:siem.notifications","endpoint:user-artifact-packager","security:endpoint-diagnostics","security:endpoint-meta-telemetry","security:telemetry-lists","security:telemetry-detection-rules","security:telemetry-prebuilt-rule-alerts","security:telemetry-timelines","security:telemetry-configuration","security:telemetry-filterlist-artifact","endpoint:metadata-check-transforms-task","alerting:metrics.alert.anomaly","alerting:logs.alert.document.count","alerting:metrics.alert.inventory.threshold","alerting:metrics.alert.threshold","alerting:monitoring_alert_cluster_health","alerting:monitoring_alert_license_expiration","alerting:monitoring_alert_cpu_usage","alerting:monitoring_alert_missing_monitoring_data","alerting:monitoring_alert_disk_usage","alerting:monitoring_alert_thread_pool_search_rejections","alerting:monitoring_alert_thread_pool_write_rejections","alerting:monitoring_alert_jvm_memory_usage","alerting:monitoring_alert_nodes_changed","alerting:monitoring_alert_logstash_version_mismatch","alerting:monitoring_alert_kibana_version_mismatch","alerting:monitoring_alert_elasticsearch_version_mismatch","alerting:monitoring_ccr_read_exceptions","alerting:monitoring_shard_size","apm-telemetry-task","alerting:apm.transaction_duration","alerting:apm.anomaly","alerting:apm.error_rate","alerting:apm.transaction_error_rate"],"unusedTaskTypes":["sampleTaskRemovedType","alerting:siem.signals","search_sessions_monitor","search_sessions_cleanup","search_sessions_expire"],"taskMaxAttempts":{"apm-source-map-migration-task":5}}},"sort":[{"_script":{"type":"number","order":"asc","script":{"lang":"painless","source":"\nif (doc['task.retryAt'].size()!=0) {\n return doc['task.retryAt'].value.toInstant().toEpochMilli();\n}\nif (doc['task.runAt'].size()!=0) {\n return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-08-18T14:21:23.169+02:00][DEBUG][plugins.monitoring.monitoring.kibana-monitoring] not sending [kibana_settings] monitoring document because [undefined] is null or invalid.
[2023-08-18T14:21:23.170+02:00][DEBUG][plugins.monitoring.monitoring.kibana-monitoring] Uploading bulk stats payload to the local cluster
[2023-08-18T14:21:23.175+02:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n if (params.claimableTaskTypes.contains(ctx._source.task.taskType)) {\n if (ctx._source.task.schedule != null || ctx._source.task.attempts < params.taskMaxAttempts[ctx._source.task.taskType]) {\n if(ctx._source.task.retryAt != null && ZonedDateTime.parse(ctx._source.task.retryAt).toInstant().toEpochMilli() < params.now) {\n ctx._source.task.scheduledAt=ctx._source.task.retryAt;\n } else {\n ctx._source.task.scheduledAt=ctx._source.task.runAt;\n }\n ctx._source.task.status = \"claiming\"; ctx._source.task.ownerId=params.fieldUpdates.ownerId; ctx._source.task.retryAt=params.fieldUpdates.retryAt;\n } else {\n ctx._source.task.status = \"failed\";\n }\n } else if (params.unusedTaskTypes.contains(ctx._source.task.taskType)) {\n ctx._source.task.status = \"unrecognized\";\n } else {\n ctx.op = \"noop\";\n }","lang":"painless","params":{"now":1692361283100,"fieldUpdates":{"ownerId":"kibana:3cc50870-5b0a-4625-a563-055064a00ef7","retryAt":"2023-08-18T12:21:53.090Z"},"claimableTaskTypes":["session_cleanup","actions_telemetry","cleanup_failed_action_executions","alerting_telemetry","alerts_invalidate_api_keys","alerting_health_check","reports:monitor","alerting:transform_health","actions:.email","actions:.index","actions:.pagerduty","actions:.swimlane","actions:.server-log","actions:.slack","actions:.webhook","actions:.cases-webhook","actions:.xmatters","actions:.servicenow","actions:.servicenow-sir","actions:.servicenow-itom","actions:.jira","actions:.resilient","actions:.teams","actions:.torq","actions:.opsgenie","actions:.tines","alerting:.index-threshold","alerting:.geo-containment","alerting:.es-query","dashboard_telemetry","cases-telemetry-task","Fleet-Usage-Sender","Fleet-Usage-Logger","fleet:reassign_action:retry","fleet:unenroll_action:retry","fleet:upgrade_action:retry","fleet:update_agent_tags:retry","fleet:request_diagnostics:retry","fleet:check-deleted-files-task","osquery:telemetry-packs","osquery:telemetry-saved-queries","osquery:telemetry-configs","cloud_security_posture-stats_task","ML:saved-objects-sync","alerting:xpack.ml.anomaly_detection_alert","alerting:xpack.ml.anomaly_detection_jobs_health","UPTIME:SyntheticsService:Sync-Saved-Monitor-Objects","alerting:xpack.uptime.alerts.monitorStatus","alerting:xpack.uptime.alerts.tlsCertificate","alerting:xpack.uptime.alerts.durationAnomaly","alerting:xpack.uptime.alerts.tls","alerting:xpack.synthetics.alerts.monitorStatus","alerting:siem.eqlRule","alerting:siem.savedQueryRule","alerting:siem.indicatorRule","alerting:siem.mlRule","alerting:siem.queryRule","alerting:siem.thresholdRule","alerting:siem.newTermsRule","alerting:siem.notifications","endpoint:user-artifact-packager","security:endpoint-diagnostics","security:endpoint-meta-telemetry","security:telemetry-lists","security:telemetry-detection-rules","security:telemetry-prebuilt-rule-alerts","security:telemetry-timelines","security:telemetry-configuration","security:telemetry-filterlist-artifact","endpoint:metadata-check-transforms-task","alerting:metrics.alert.anomaly","alerting:logs.alert.document.count","alerting:metrics.alert.inventory.threshold","alerting:metrics.alert.threshold","alerting:monitoring_alert_cluster_health","alerting:monitoring_alert_license_expiration","alerting:monitoring_alert_cpu_usage","alerting:monitoring_alert_missing_monitoring_data","alerting:monitoring_alert_disk_usage","alerting:monitoring_alert_thread_pool_search_rejections","alerting:monitoring_alert_thread_pool_write_rejections","alerting:monitoring_alert_jvm_memory_usage","alerting:monitoring_alert_nodes_changed","alerting:monitoring_alert_logstash_version_mismatch","alerting:monitoring_alert_kibana_version_mismatch","alerting:monitoring_alert_elasticsearch_version_mismatch","alerting:monitoring_ccr_read_exceptions","alerting:monitoring_shard_size","apm-telemetry-task","alerting:apm.transaction_duration","alerting:apm.anomaly","alerting:apm.error_rate","alerting:apm.transaction_error_rate"],"skippedTaskTypes":["report:execute","apm-source-map-migration-task"],"unusedTaskTypes":["sampleTaskRemovedType","alerting:siem.signals","search_sessions_monitor","search_sessions_cleanup","search_sessions_expire"],"taskMaxAttempts":{"session_cleanup":3,"actions_telemetry":3,"cleanup_failed_action_executions":3,"alerting_telemetry":3,"alerts_invalidate_api_keys":3,"alerting_health_check":3,"reports:monitor":1,"alerting:transform_health":3,"actions:.email":3,"actions:.index":3,"actions:.pagerduty":3,"actions:.swimlane":3,"actions:.server-log":3,"actions:.slack":3,"actions:.webhook":3,"actions:.cases-webhook":3,"actions:.xmatters":3,"actions:.servicenow":3,"actions:.servicenow-sir":3,"actions:.servicenow-itom":3,"actions:.jira":3,"actions:.resilient":3,"actions:.teams":3,"actions:.torq":3,"actions:.opsgenie":3,"actions:.tines":3,"alerting:.index-threshold":3,"alerting:.geo-containment":3,"alerting:.es-query":3,"dashboard_telemetry":3,"cases-telemetry-task":3,"Fleet-Usage-Sender":1,"Fleet-Usage-Logger":1,"fleet:reassign_action:retry":1,"fleet:unenroll_action:retry":1,"fleet:upgrade_action:retry":1,"fleet:update_agent_tags:retry":1,"fleet:request_diagnostics:retry":1,"fleet:check-deleted-files-task":3,"osquery:telemetry-packs":3,"osquery:telemetry-saved-queries":3,"osquery:telemetry-configs":3,"cloud_security_posture-stats_task":3,"ML:saved-objects-sync":3,"alerting:xpack.ml.anomaly_detection_alert":3,"alerting:xpack.ml.anomaly_detection_jobs_health":3,"UPTIME:SyntheticsService:Sync-Saved-Monitor-Objects":3,"alerting:xpack.uptime.alerts.monitorStatus":3,"alerting:xpack.uptime.alerts.tlsCertificate":3,"alerting:xpack.uptime.alerts.durationAnomaly":3,"alerting:xpack.uptime.alerts.tls":3,"alerting:xpack.synthetics.alerts.monitorStatus":3,"alerting:siem.eqlRule":3,"alerting:siem.savedQueryRule":3,"alerting:siem.indicatorRule":3,"alerting:siem.mlRule":3,"alerting:siem.queryRule":3,"alerting:siem.thresholdRule":3,"alerting:siem.newTermsRule":3,"alerting:siem.notifications":3,"endpoint:user-artifact-packager":3,"security:endpoint-diagnostics":3,"security:endpoint-meta-telemetry":3,"security:telemetry-lists":3,"security:telemetry-detection-rules":3,"security:telemetry-prebuilt-rule-alerts":3,"security:telemetry-timelines":3,"security:telemetry-configuration":3,"security:telemetry-filterlist-artifact":3,"endpoint:metadata-check-transforms-task":3,"alerting:metrics.alert.anomaly":3,"alerting:logs.alert.document.count":3,"alerting:metrics.alert.inventory.threshold":3,"alerting:metrics.alert.threshold":3,"alerting:monitoring_alert_cluster_health":3,"alerting:monitoring_alert_license_expiration":3,"alerting:monitoring_alert_cpu_usage":3,"alerting:monitoring_alert_missing_monitoring_data":3,"alerting:monitoring_alert_disk_usage":3,"alerting:monitoring_alert_thread_pool_search_rejections":3,"alerting:monitoring_alert_thread_pool_write_rejections":3,"alerting:monitoring_alert_jvm_memory_usage":3,"alerting:monitoring_alert_nodes_changed":3,"alerting:monitoring_alert_logstash_version_mismatch":3,"alerting:monitoring_alert_kibana_version_mismatch":3,"alerting:monitoring_alert_elasticsearch_version_mismatch":3,"alerting:monitoring_ccr_read_exceptions":3,"alerting:monitoring_shard_size":3,"apm-telemetry-task":3,"alerting:apm.transaction_duration":3,"alerting:apm.anomaly":3,"alerting:apm.error_rate":3,"alerting:apm.transaction_error_rate":3}}},"sort":[{"_script":{"type":"number","order":"asc","script":{"lang":"painless","source":"\nif (doc['task.retryAt'].size()!=0) {\n return doc['task.retryAt'].value.toInstant().toEpochMilli();\n}\nif (doc['task.runAt'].size()!=0) {\n return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n "}}}],"max_docs":10,"conflicts":"proceed"}
[2023-08-18T14:21:23.182+02:00][DEBUG][elasticsearch.query.data] 200 - 898.0B
POST /.kibana_task_manager/_search?ignore_unavailable=true
{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"term":{"task.ownerId":"kibana:3cc50870-5b0a-4625-a563-055064a00ef7"}},{"term":{"task.status":"claiming"}},{"bool":{"should":[{"term":{"task.taskType":"session_cleanup"}},{"term":{"task.taskType":"actions_telemetry"}},{"term":{"task.taskType":"cleanup_failed_action_executions"}},{"term":{"task.taskType":"alerting_telemetry"}},{"term":{"task.taskType":"alerts_invalidate_api_keys"}},{"term":{"task.taskType":"alerting_health_check"}},{"term":{"task.taskType":"reports:monitor"}},{"term":{"task.taskType":"alerting:transform_health"}},{"term":{"task.taskType":"actions:.email"}},{"term":{"task.taskType":"actions:.index"}},{"term":{"task.taskType":"actions:.pagerduty"}},{"term":{"task.taskType":"actions:.swimlane"}},{"term":{"task.taskType":"actions:.server-log"}},{"term":{"task.taskType":"actions:.slack"}},{"term":{"task.taskType":"actions:.webhook"}},{"term":{"task.taskType":"actions:.cases-webhook"}},{"term":{"task.taskType":"actions:.xmatters"}},{"term":{"task.taskType":"actions:.servicenow"}},{"term":{"task.taskType":"actions:.servicenow-sir"}},{"term":{"task.taskType":"actions:.servicenow-itom"}},{"term":{"task.taskType":"actions:.jira"}},{"term":{"task.taskType":"actions:.resilient"}},{"term":{"task.taskType":"actions:.teams"}},{"term":{"task.taskType":"actions:.torq"}},{"term":{"task.taskType":"actions:.opsgenie"}},{"term":{"task.taskType":"actions:.tines"}},{"term":{"task.taskType":"alerting:.index-threshold"}},{"term":{"task.taskType":"alerting:.geo-containment"}},{"term":{"task.taskType":"alerting:.es-query"}},{"term":{"task.taskType":"dashboard_telemetry"}},{"term":{"task.taskType":"cases-telemetry-task"}},{"term":{"task.taskType":"Fleet-Usage-Sender"}},{"term":{"task.taskType":"Fleet-Usage-Logger"}},{"term":{"task.taskType":"fleet:reassign_action:retry"}},{"term":{"task.taskType":"fleet:unenroll_action:retry"}},{"term":{"task.taskType":"fleet:upgrade_action:retry"}},{"term":{"task.taskType":"fleet:update_agent_tags:retry"}},{"term":{"task.taskType":"fleet:request_diagnostics:retry"}},{"term":{"task.taskType":"fleet:check-deleted-files-task"}},{"term":{"task.taskType":"osquery:telemetry-packs"}},{"term":{"task.taskType":"osquery:telemetry-saved-queries"}},{"term":{"task.taskType":"osquery:telemetry-configs"}},{"term":{"task.taskType":"cloud_security_posture-stats_task"}},{"term":{"task.taskType":"ML:saved-objects-sync"}},{"term":{"task.taskType":"alerting:xpack.ml.anomaly_detection_alert"}},{"term":{"task.taskType":"alerting:xpack.ml.anomaly_detection_jobs_health"}},{"term":{"task.taskType":"UPTIME:SyntheticsService:Sync-Saved-Monitor-Objects"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.monitorStatus"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.tlsCertificate"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.durationAnomaly"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.tls"}},{"term":{"task.taskType":"alerting:xpack.synthetics.alerts.monitorStatus"}},{"term":{"task.taskType":"alerting:siem.eqlRule"}},{"term":{"task.taskType":"alerting:siem.savedQueryRule"}},{"term":{"task.taskType":"alerting:siem.indicatorRule"}},{"term":{"task.taskType":"alerting:siem.mlRule"}},{"term":{"task.taskType":"alerting:siem.queryRule"}},{"term":{"task.taskType":"alerting:siem.thresholdRule"}},{"term":{"task.taskType":"alerting:siem.newTermsRule"}},{"term":{"task.taskType":"alerting:siem.notifications"}},{"term":{"task.taskType":"endpoint:user-artifact-packager"}},{"term":{"task.taskType":"security:endpoint-diagnostics"}},{"term":{"task.taskType":"security:endpoint-meta-telemetry"}},{"term":{"task.taskType":"security:telemetry-lists"}},{"term":{"task.taskType":"security:telemetry-detection-rules"}},{"term":{"task.taskType":"security:telemetry-prebuilt-rule-alerts"}},{"term":{"task.taskType":"security:telemetry-timelines"}},{"term":{"task.taskType":"security:telemetry-configuration"}},{"term":{"task.taskType":"security:telemetry-filterlist-artifact"}},{"term":{"task.taskType":"endpoint:metadata-check-transforms-task"}},{"term":{"task.taskType":"alerting:metrics.alert.anomaly"}},{"term":{"task.taskType":"alerting:logs.alert.document.count"}},{"term":{"task.taskType":"alerting:metrics.alert.inventory.threshold"}},{"term":{"task.taskType":"alerting:metrics.alert.threshold"}},{"term":{"task.taskType":"alerting:monitoring_alert_cluster_health"}},{"term":{"task.taskType":"alerting:monitoring_alert_license_expiration"}},{"term":{"task.taskType":"alerting:monitoring_alert_cpu_usage"}},{"term":{"task.taskType":"alerting:monitoring_alert_missing_monitoring_data"}},{"term":{"task.taskType":"alerting:monitoring_alert_disk_usage"}},{"term":{"task.taskType":"alerting:monitoring_alert_thread_pool_search_rejections"}},{"term":{"task.taskType":"alerting:monitoring_alert_thread_pool_write_rejections"}},{"term":{"task.taskType":"alerting:monitoring_alert_jvm_memory_usage"}},{"term":{"task.taskType":"alerting:monitoring_alert_nodes_changed"}},{"term":{"task.taskType":"alerting:monitoring_alert_logstash_version_mismatch"}},{"term":{"task.taskType":"alerting:monitoring_alert_kibana_version_mismatch"}},{"term":{"task.taskType":"alerting:monitoring_alert_elasticsearch_version_mismatch"}},{"term":{"task.taskType":"alerting:monitoring_ccr_read_exceptions"}},{"term":{"task.taskType":"alerting:monitoring_shard_size"}},{"term":{"task.taskType":"apm-telemetry-task"}},{"term":{"task.taskType":"alerting:apm.transaction_duration"}},{"term":{"task.taskType":"alerting:apm.anomaly"}},{"term":{"task.taskType":"alerting:apm.error_rate"}},{"term":{"task.taskType":"alerting:apm.transaction_error_rate"}}]}}]}}]}},"size":10,"seq_no_primary_term":true,"sort":{"_script":{"type":"number","order":"asc","script":{"lang":"painless","source":"\nif (doc['task.retryAt'].size()!=0) {\n return doc['task.retryAt'].value.toInstant().toEpochMilli();\n}\nif (doc['task.runAt'].size()!=0) {\n return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n "}}}}
After that WinlogBeat ran without problems and the messages were visible in Kibana.
The last thing I did was to start Metricbeat. Of course, I received an error message there as well
{"log.level":"error","@timestamp":"2023-08-18T15:36:00.079+0200","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":120},"message":"Dropping encrypted pem 'RSA PRIVATE KEY' block read from C:\\_Mon\\m\\certs\\http-wxtask2p.idm.lan.local.key. no passphrase available","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-08-18T15:36:00.080+0200","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":59},"message":"Failed reading key file: no PEM blocks","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-18T15:36:00.080+0200","log.origin":{"file.name":"instance/beat.go","file.line":442},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-08-18T15:36:00.081+0200","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: 1 error: no PEM blocks C:\\_Mon\\m\\certs\\http-wxtask2p.idm.lan.local.key accessing 'output.elasticsearch' (source:'metricbeat.yml')","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error initializing publisher: 1 error: no PEM blocks C:\_Mon\m\certs\http-wxtask2p.idm.lan.local.key accessing 'output.elasticsearch' (source:'metricbeat.yml')
I then solved an error in metricbeat.yml and got another message:
{"log.level":"error","@timestamp":"2023-08-18T16:03:10.181+0200","log.origin":{"file.name":"cfgfile/reload.go","file.line":273},"message":"Error loading config from file 'C:\\_Mon\\m\\modules.d\\elasticsearch-xpack.yml', error invalid config: yaml: line 9: did not find expected '-' indicator","service.name":"metricbeat","ecs.version":"1.6.0"}
After updating this file as well, metricbeat worked again.
However, I also had to modify the metricbeat.yml files on the devices I wanted to monitor.
And then it was done...
One problem that exists at the moment is that I had to use the username and password entries very often. This data is currently in plain text in the config files. This isn´t so nice..
How can I make this more secure? I get an error in some applications when I try to use the keystore..!?
I would check this again next week and get back to you.
Or should I open a separate thread for this?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
With all you know now, I'm wondering if it'd make sense to start with 
