Kibana stopped showing logs; error failed to poll for work

Elastic Stack Kibana
1

Hello there,

I recently have some trouble with my ELK stack which suddently stoppend working since 2022 july 4th.
With some research in the logs, i've found somehting strange :

{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:22:35.344+02:00","message":"Kibana is now unavailable (was available)","log":{"level":"INFO","logger":"status"},"process":{"pid":316111},"trace":{"id":"42291becce105f7fc0226f91ee0860ba"},"transaction":{"id":"1eebd6f88af0b827"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:22:38.193+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"65c9b11dc07c51fa60e497a75ccf3b80"},"transaction":{"id":"2f8f348e94c3a87d"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:22:38.423+02:00","message":"Kibana is now degraded (was unavailable)","log":{"level":"INFO","logger":"status"},"process":{"pid":316111},"trace":{"id":"8d4332e1d5b57b9597b1f3cecce8c6bd"},"transaction":{"id":"83f1afb7f348bdd4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:23:08.197+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"8d4332e1d5b57b9597b1f3cecce8c6bd"},"transaction":{"id":"83f1afb7f348bdd4"}}

I can't figure out what happend, and i don't know where to find reveland logs from elasticsearch to find where did the problem come.

Can anyone help me ?

2

Hi @Baguette. There is not enough information here to troubleshoot. I think we need more logs from Kibana or logs from Elasticsearch. How did you install and start Elasticsearch and Kibana?

3

Hello @nickpeihl,

Elasticsearch, logstsh and kibana were installed via apt repository on debian 10. Kibana and elasticsearch were upgrade to v 8.0 last month.
I can give more kibana logs; but for elasticsearch, which log file did you need ? all ?

4

Based on the documentation the elasticsearch logs should be in /var/log/elasticsearch/{your-cluster-name}.log. Please check and report if there are any errors in that file.

5

In term of kibana logs, i found nothing relevant; like that

{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-03T21:09:00.149+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"3bd41430d94dc913"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-03T21:38:54.347+02:00","message":"no endpoint installation found","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1"},"process":{"pid":316111},"trace":{"id":"3f1ee434df75db4c4b6dfcc65d2d2738"},"transaction":{"id":"0fe53d2f7825fb79"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-03T22:09:00.973+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"3502cf52974a2090"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-03T23:09:03.654+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"118a3a4a930edfe9"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-03T23:38:54.637+02:00","message":"no endpoint installation found","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1"},"process":{"pid":316111},"trace":{"id":"3f1ee434df75db4c4b6dfcc65d2d2738"},"transaction":{"id":"52009db8201a33ce"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T00:00:01.326+02:00","message":"Error executing alerting telemetry task: ResponseError: search_phase_execution_exception: ","log":{"level":"WARN","logger":"plugins.alerting.usage"},"process":{"pid":316111},"trace":{"id":"3f1ee434df75db4c4b6dfcc65d2d2738"},"transaction":{"id":"7997469b7b42e42c"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T00:09:06.794+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"71ab877f0528cf95"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T01:09:06.945+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"9da5b8b678b92bf5"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T01:38:55.045+02:00","message":"no endpoint installation found","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1"},"process":{"pid":316111},"trace":{"id":"3f1ee434df75db4c4b6dfcc65d2d2738"},"transaction":{"id":"307f2ba1ac7118fa"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T02:09:07.136+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"f9964b484e0fa6b8"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T03:09:07.436+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"410d83685b97be24"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T03:38:55.592+02:00","message":"no endpoint installation found","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1"},"process":{"pid":316111},"trace":{"id":"3f1ee434df75db4c4b6dfcc65d2d2738"},"transaction":{"id":"cc09a4e6297818a6"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T04:09:07.671+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"17422b1411825a09"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T05:09:07.777+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"0734372044a3d0f5"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T05:38:55.801+02:00","message":"no endpoint installation found","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1"},"process":{"pid":316111},"trace":{"id":"3f1ee434df75db4c4b6dfcc65d2d2738"},"transaction":{"id":"b0b89bde3c4ef849"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T06:09:08.150+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"3dc8157876f43e51"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T07:09:08.198+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"f7d06c8523155ac8"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T07:38:56.273+02:00","message":"no endpoint installation found","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1"},"process":{"pid":316111},"trace":{"id":"3f1ee434df75db4c4b6dfcc65d2d2738"},"transaction":{"id":"b9ee106e5b8e0dfb"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T08:09:08.436+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"8a9102f7c91244ab"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:09:11.643+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":316111},"span":{"id":"b155134a412763f7"},"trace":{"id":"e3656255a11766b63523f03463d882d4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:22:35.344+02:00","message":"Kibana is now unavailable (was available)","log":{"level":"INFO","logger":"status"},"process":{"pid":316111},"trace":{"id":"42291becce105f7fc0226f91ee0860ba"},"transaction":{"id":"1eebd6f88af0b827"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:22:38.193+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"65c9b11dc07c51fa60e497a75ccf3b80"},"transaction":{"id":"2f8f348e94c3a87d"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:22:38.423+02:00","message":"Kibana is now degraded (was unavailable)","log":{"level":"INFO","logger":"status"},"process":{"pid":316111},"trace":{"id":"8d4332e1d5b57b9597b1f3cecce8c6bd"},"transaction":{"id":"83f1afb7f348bdd4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:23:08.197+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"8d4332e1d5b57b9597b1f3cecce8c6bd"},"transaction":{"id":"83f1afb7f348bdd4"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:23:38.201+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"e17e4f7002f0d41b91d129c0ae67ba9f"},"transaction":{"id":"d8bbad23b2cf317b"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:24:08.205+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"7591e7149ab98a6fb9dcfaebbbd7585c"},"transaction":{"id":"79f4e1381e820c53"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:24:38.209+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"be435f1062104cc86de5e116a39c52ef"},"transaction":{"id":"601aad0d3ec07e14"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:25:08.211+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"4895d441a65d32003d5316820087a6ea"},"transaction":{"id":"e168488b78535bda"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:25:38.217+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"74a96a17b3792baf7587f57bd8410d50"},"transaction":{"id":"5a9de7fdf821466b"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:26:08.220+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"a53153783750ef3428755cdb91bf25de"},"transaction":{"id":"c188ba331d5da047"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:26:38.224+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"914e3aa35c0029561c2e1297e48e34ef"},"transaction":{"id":"bae7bafe912182af"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:27:08.227+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"9fc7fe997a986bcdfc1bfee4fc4fd1f1"},"transaction":{"id":"059d416de927cfdf"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:27:38.229+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"31d53866e8d184d8a7fbdcd6b63b104c"},"transaction":{"id":"b7263a82f7ac043e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:28:08.233+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"6a6a53b508f0aa3d8de2dbd46ca47479"},"transaction":{"id":"f0a0d6760681a97f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-04T09:28:38.241+02:00","message":"Failed to poll for work: Error: work has timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":316111},"trace":{"id":"d136c02fce9d262ff9771a7379c0c375"},"transaction":{"id":"5e0ac878a3e39265"}}

For elasticsearch, the only logs I have are about treshold on disk usage ,which was 95% full

6

By default Elasticsearch will stop writing to indices when you've reached flood stage watermark (95% disk usage). Kibana continually writes to an internal Elasticsearch index and Kibana will not start if the index is not-writable. I suggest taking your disk usage to below 90% and see if that resolves the issue.

7

Unfortunally, this does not resolve the issue;

Here is a screen capture of my disc usage and kibana interface
image

image
image1370×710 185 KB

8

Is Elasticsearch still warning about disk usage? Have you restarted Elasticsearch and Kibana?

9

Hello,

I have no issue with elasticsearch, as you can see in the capture bellow

image
image1422×578 25.6 KB

For kibana : here is what i've found in the logs

{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.111+02:00","message":"connect ECONNREFUSED 127.0.0.1:9200","error":{"message":"connect ECONNREFUSED 127.0.0.1:9200","type":"ConnectionError","stack_trace":"ConnectionError: connect ECONNREFUSED 127.0.0.1:9200\n    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at KibanaTransport.request (/usr/share/kibana/src/core/server/elasticsearch/client/create_transport.js:58:16)\n    at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12)\n    at SessionIndex.get (/usr/share/kibana/x-pack/plugins/security/server/session_management/session_index.js:129:11)\n    at Session.get (/usr/share/kibana/x-pack/plugins/security/server/session_management/session.js:89:31)\n    at Authenticator.getSessionValue (/usr/share/kibana/x-pack/plugins/security/server/authentication/authenticator.js:461:34)\n    at Authenticator.authenticate (/usr/share/kibana/x-pack/plugins/security/server/authentication/authenticator.js:259:34)\n    at /usr/share/kibana/x-pack/plugins/security/server/authentication/authentication_service.js:87:36\n    at Object.interceptAuth [as authenticate] (/usr/share/kibana/src/core/server/http/lifecycle/auth.js:90:22)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at module.exports.internals.Auth._authenticate (/usr/share/kibana/node_modules/@hapi/hapi/lib/auth.js:273:30)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"},"log":{"level":"ERROR","logger":"http.server.Kibana"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.207+02:00","message":"License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:9200 error","log":{"level":"WARN","logger":"plugins.licensing"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.211+02:00","message":"You cannot use searchprofiler because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.searchprofiler"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.220+02:00","message":"You cannot use painlessLab because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.painlessLab"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.225+02:00","message":"You cannot use snapshot_restore because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.snapshotRestore"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.227+02:00","message":"You cannot use transform because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.transform"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.230+02:00","message":"You cannot use Remote Clusters because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.remoteClusters"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.234+02:00","message":"You cannot use index_lifecycle_management because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.indexLifecycleManagement"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.235+02:00","message":"You cannot use rollup because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.rollup"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.242+02:00","message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:07.277+02:00","message":"Unable to verify endpoint policies in line with license change: failed to fetch package policies: connect ECONNREFUSED 127.0.0.1:9200","log":{"level":"WARN","logger":"plugins.securitySolution"},"process":{"pid":4506},"trace":{"id":"4a117f46594b8a1ec0fe7ca81dc8b38e"},"transaction":{"id":"0d7d7d3678df1aae"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:14.247+02:00","message":"License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:9200 error","log":{"level":"WARN","logger":"plugins.licensing"},"process":{"pid":4506},"trace":{"id":"1edef660d993a0acb3edcfe0ea138007"},"transaction":{"id":"214ea38fd9be719f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:22.989+02:00","message":"License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:9200 error","log":{"level":"WARN","logger":"plugins.licensing"},"process":{"pid":4506},"trace":{"id":"5c8b01f65cb6fdb1607620b4d8069feb"},"transaction":{"id":"3f0c30fae26b4f1f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:22.998+02:00","message":"X-Pack Monitoring Cluster Alerts will not be available: connect ECONNREFUSED 127.0.0.1:9200","log":{"level":"WARN","logger":"plugins.monitoring.monitoring"},"process":{"pid":4506},"trace":{"id":"5c8b01f65cb6fdb1607620b4d8069feb"},"transaction":{"id":"3f0c30fae26b4f1f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:44.259+02:00","message":"License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:9200 error","log":{"level":"WARN","logger":"plugins.licensing"},"process":{"pid":4506},"trace":{"id":"1edef660d993a0acb3edcfe0ea138007"},"transaction":{"id":"214ea38fd9be719f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:53.506+02:00","message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":4506},"trace":{"id":"1edef660d993a0acb3edcfe0ea138007"},"transaction":{"id":"214ea38fd9be719f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:53.580+02:00","message":"License information could not be obtained from Elasticsearch due to ResponseError: security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_xpack] error","log":{"level":"WARN","logger":"plugins.licensing"},"process":{"pid":4506},"trace":{"id":"5c8b01f65cb6fdb1607620b4d8069feb"},"transaction":{"id":"3f0c30fae26b4f1f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:53.582+02:00","message":"X-Pack Monitoring Cluster Alerts will not be available: security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_xpack]","log":{"level":"WARN","logger":"plugins.monitoring.monitoring"},"process":{"pid":4506},"trace":{"id":"5c8b01f65cb6fdb1607620b4d8069feb"},"transaction":{"id":"3f0c30fae26b4f1f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:50:57.633+02:00","message":"Kibana is now unavailable (was critical)","log":{"level":"INFO","logger":"status"},"process":{"pid":4506},"trace":{"id":"1edef660d993a0acb3edcfe0ea138007"},"transaction":{"id":"214ea38fd9be719f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:51:14.377+02:00","message":"Starting monitoring stats collection","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":4506},"trace":{"id":"1edef660d993a0acb3edcfe0ea138007"},"transaction":{"id":"214ea38fd9be719f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:51:14.461+02:00","message":"Kibana is now available (was unavailable)","log":{"level":"INFO","logger":"status"},"process":{"pid":4506},"trace":{"id":"1edef660d993a0acb3edcfe0ea138007"},"transaction":{"id":"214ea38fd9be719f"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:52:23.853+02:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":4506}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:52:23.861+02:00","message":"Monitoring stats collection is stopped","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":4506}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:02.532+02:00","message":"Plugin \"cloudSecurityPosture\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":6925},"trace":{"id":"9a45d4eab68117e7a6ccf02f592e61c5"},"transaction":{"id":"ff31379efd7badc3"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:02.581+02:00","message":"Plugin \"metricsEntities\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":6925},"trace":{"id":"9a45d4eab68117e7a6ccf02f592e61c5"},"transaction":{"id":"ff31379efd7badc3"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:02.676+02:00","message":"http server running at http://localhost:5601/kibana","log":{"level":"INFO","logger":"http.server.Preboot"},"process":{"pid":6925},"trace":{"id":"9a45d4eab68117e7a6ccf02f592e61c5"},"transaction":{"id":"ff31379efd7badc3"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:02.762+02:00","message":"Setting up [1] plugins: [interactiveSetup]","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":6925},"trace":{"id":"9a45d4eab68117e7a6ccf02f592e61c5"},"transaction":{"id":"ff31379efd7badc3"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:02.890+02:00","message":"The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set \"xpack.reporting.roles.enabled\" to \"false\" to adopt the future behavior before upgrading.","log":{"level":"WARN","logger":"config.deprecation"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:03.616+02:00","message":"Setting up [117] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,screenshotting,banners,newsfeed,fieldFormats,expressions,eventAnnotation,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,advancedSettings,spaces,security,savedObjectsTagging,reporting,lists,fileUpload,ingestPipelines,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,telemetry,licenseManagement,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,savedObjectsManagement,console,controls,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,dataViewFieldEditor,sharedUX,discover,lens,osquery,maps,dataVisualizer,ml,cases,timelines,sessionView,securitySolution,observability,uptime,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,discoverEnhanced,dataViewManagement]","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:03.638+02:00","message":"TaskManager is identified by the Kibana UUID: 8f931ad8-9db7-45fe-bf2e-e68b5aafaa8e","log":{"level":"INFO","logger":"plugins.taskManager"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:03.896+02:00","message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:03.898+02:00","message":"Session cookies will be transmitted over insecure connections. This is not recommended.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:03.920+02:00","message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:03.921+02:00","message":"Session cookies will be transmitted over insecure connections. This is not recommended.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:03.966+02:00","message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.reporting.config"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:03.983+02:00","message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.encryptedSavedObjects"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:04.004+02:00","message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.actions"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:04.021+02:00","message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.alerting"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:04.060+02:00","message":"Installing common resources shared between all indices","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:05.230+02:00","message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Debian 11 OS. Automatically setting 'xpack.screenshotting.capture.browser.chromium.disableSandbox: true'.","log":{"level":"WARN","logger":"plugins.screenshotting.config"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:05.593+02:00","message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:05.597+02:00","message":"Starting saved objects migrations","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:05.825+02:00","message":"[.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 208ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:05.940+02:00","message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 316ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:05.972+02:00","message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 147ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:06.084+02:00","message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 144ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:06.391+02:00","message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 307ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:06.432+02:00","message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 460ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:06.439+02:00","message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 48ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:06.487+02:00","message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 55ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:06.834+02:00","message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 395ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:07.217+02:00","message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 383ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:07.218+02:00","message":"[.kibana_task_manager] Migration completed after 1594ms","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:07.259+02:00","message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 772ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:10.024+02:00","message":"[.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 2765ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:10.041+02:00","message":"[.kibana] Migration completed after 4424ms","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:10.109+02:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:10.127+02:00","message":"Starting [117] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,screenshotting,banners,newsfeed,fieldFormats,expressions,eventAnnotation,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,advancedSettings,spaces,security,savedObjectsTagging,reporting,lists,fileUpload,ingestPipelines,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,telemetry,licenseManagement,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,savedObjectsManagement,console,controls,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,dataViewFieldEditor,sharedUX,discover,lens,osquery,maps,dataVisualizer,ml,cases,timelines,sessionView,securitySolution,observability,uptime,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,discoverEnhanced,dataViewManagement]","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:12.987+02:00","message":"config sourced from: production cluster","log":{"level":"INFO","logger":"plugins.monitoring.monitoring"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.244+02:00","message":"http server running at http://localhost:5601/kibana","log":{"level":"INFO","logger":"http.server.Kibana"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.299+02:00","message":"Kibana is now degraded","log":{"level":"INFO","logger":"status"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.640+02:00","message":"Starting monitoring stats collection","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.641+02:00","message":"Beginning fleet setup","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.727+02:00","message":"Installed common resources shared between all indices","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.732+02:00","message":"Installing resources for index .alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.734+02:00","message":"Installing resources for index .preview.alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.734+02:00","message":"Installing resources for index .alerts-observability.uptime.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.735+02:00","message":"Installing resources for index .alerts-observability.logs.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.735+02:00","message":"Installing resources for index .alerts-observability.metrics.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.746+02:00","message":"Installing resources for index .alerts-observability.apm.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.926+02:00","message":"Installed resources for index .alerts-observability.apm.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.928+02:00","message":"Installed resources for index .alerts-observability.metrics.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.932+02:00","message":"Installed resources for index .alerts-observability.logs.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.934+02:00","message":"Installed resources for index .alerts-observability.uptime.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:15.940+02:00","message":"Installed resources for index .alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:16.056+02:00","message":"Installed resources for index .preview.alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:16.128+02:00","message":"Task ML:saved-objects-sync-task: scheduled with interval 1h","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:16.263+02:00","message":"Fleet setup completed","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:16.274+02:00","message":"Dependent plugin setup complete - Starting ManifestTask","log":{"level":"INFO","logger":"plugins.securitySolution"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"60f170206ec07626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:16.307+02:00","message":"Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:16.310+02:00","message":"Enabling the Chromium sandbox provides an additional layer of protection.","log":{"level":"WARN","logger":"plugins.screenshotting.chromium"},"process":{"pid":6925},"trace":{"id":"89908c79bf9461745ced9b9c52883226"},"transaction":{"id":"215937f9ac11bf2e"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:18.768+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"3b9d36ec5880c9d5"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:53:21.311+02:00","message":"Kibana is now available (was degraded)","log":{"level":"INFO","logger":"status"},"process":{"pid":6925},"trace":{"id":"b66333e8f6acb1cb03eae984a770cca9"},"transaction":{"id":"6ddc1936822f1626"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T17:54:18.589+02:00","message":"Logging in with provider \"basic\" (basic)","log":{"level":"INFO","logger":"plugins.security.routes"},"process":{"pid":6925},"trace":{"id":"d552c1286677ba6c2e31fb4acd82c417"},"transaction":{"id":"433313435cb05c3a"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T18:01:00.601+02:00","message":"no endpoint installation found","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1"},"process":{"pid":6925},"trace":{"id":"3f1ee434df75db4c4b6dfcc65d2d2738"},"transaction":{"id":"b0c959a17b87e4cb"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T18:01:54.070+02:00","message":"Kibana is now degraded (was available)","log":{"level":"INFO","logger":"status"},"process":{"pid":6925},"trace":{"id":"fa8f29813da1be898c26cb0d35732c54"},"transaction":{"id":"574494588286d5e7"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T18:02:00.450+02:00","message":"Kibana is now available (was degraded)","log":{"level":"INFO","logger":"status"},"process":{"pid":6925},"trace":{"id":"8ef37c3d4ae3e9772d478d41a90ccd01"},"transaction":{"id":"6b69a14dd50f1a14"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T18:53:18.845+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"5e0142b801fbcc5b"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-05T19:53:19.247+02:00","message":"Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization","log":{"level":"INFO","logger":"plugins.ml"},"process":{"pid":6925},"trace":{"id":"8da537f3dd0f8b8d554c4f333e52dc6d"},"transaction":{"id":"610828ef17d6f71e"}}

I see that the error ""Task ML:saved-objects-sync-task" come very often now

10

After some time, i think I find an other important detail;

First; a equipment is flooding by sending 50 messages each second

Second; some times, CPU usage on elasticsearch goes up to nearly 90% with no explanation

11

If I understand correctly, Kibana should be working now. Is that correct? If so, I believe reducing the disk usage resolved the original issue.

I see that the error ""Task ML:saved-objects-sync-task" come very often now

This appears to be informational, not an error. I this is of any concern.

After some time, i think I find an other important detail;
First; a equipment is flooding by sending 50 messages each second
Second; some times, CPU usage on elasticsearch goes up to nearly 90% with no explanation

I suggest reviewing this documentation. If you have more questions or continue to have issues with high CPU usage, I suggest opening a new discussion in the Elasticsearch forum.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.