Problem in Kibana

Elastic Stack Kibana
1

Hello everyone,

I was using ELK stack to put in place an internal SOC, all was well there was only a problem with the license but it was resolved, and then suddenky kibana stoped functioning it just gives me the error: "Kibana server is not ready yet"
i checked every solution but it doesn't work,
in the logs the error is: "Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":646778},"trace":{"id":"8afcdc9c1b23cdd5f1a463d408bd7ba4"},"transaction":{"id":"470c7a82f20ca7e2"}}"

I thank you in advance,
Best Regards,

"elasticsearch.yml:
<# ======================== Elasticsearch Configuration =========================

---------------------------------- Cluster -----------------------------------

cluster.name: cluster

------------------------------------ Node ------------------------------------

node.name: node-1

----------------------------------- Paths ------------------------------------

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

---------------------------------- Network -----------------------------------

network.host: localhost

http.port: 9200
#discovery.type: single-node

Enable security features

xpack.security.enabled: true

xpack.security.enrollment.enabled: true

Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:
enabled: true
keystore.path: /etc/elasticsearch/certs/http.p12
#verification_mode: certificate
#key: /etc/elasticsearch/certs/privkey1.pem
#certificate: /etc/elasticsearch/certs/cert1.pem

Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:
enabled: true
#verification_mode: certificate
#key: /etc/elasticsearch/certs/privkey1.pem
#certificate: /etc/elasticsearch/certs/cert1.pem
#certificate_authorities: [ "/etc/elasticsearch/certs/fullchain1.pem" ]

client_authentication: required
keystore.path: certs/elastic-certificates.p12
truststore.path: certs/elastic-certificates.p12

Create a new cluster with the current node only

Additional nodes can still join the cluster later

cluster.initial_master_nodes: ["my.soc.com"]

Allow HTTP API connections from anywhere

Connections are encrypted and require user authentication

http.host: 0.0.0.0

Allow other nodes to join the cluster from anywhere

Connections are encrypted and mutually authenticated

#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION ------------------------->

kibana.yml:

<# =================== System: Kibana Server ===================

Kibana is served by a back end server. This setting specifies the port to use.

server.port: 5601

server.host: localhost

=================== System: Kibana Server (Optional) ===================

Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.

These settings enable SSL for outgoing requests from the Kibana server to the browser.

#server.ssl.enabled: true
#server.ssl.certificate: /etc/kibana/cert1.pem
#server.ssl.key: /etc/kibana/cert1.pem

=================== System: Elasticsearch ===================

The URLs of the Elasticsearch instances to use for all your queries.

elasticsearch.hosts: ["https://localhost:9200"]

If your Elasticsearch is protected with basic authentication, these settings provide

the username and password that the Kibana server uses to perform maintenance on the Kibana

index at startup. Your Kibana users still need to authenticate with Elasticsearch, which

is proxied through the Kibana server.

#elasticsearch.username: "elastic"
#elasticsearch.password: ""

Kibana can also authenticate to Elasticsearch via "service account tokens".

Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.

Use this token instead of a username/password.

elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC15Mzc5NDU6RUdJQjNZWnpUZHFjSWlfdWZHUGN5QQ

=================== System: Elasticsearch (Optional) ===================

These files are used to verify the identity of Kibana to Elasticsearch and are required when

xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.

#elasticsearch.ssl.enabled: true
elasticsearch.ssl.certificate: /etc/kibana/fullchain1.pem
elasticsearch.ssl.key: /etc/kibana/privkey1.pem

Enables you to specify a path to the PEM file for the certificate

authority for your Elasticsearch instance.

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elasticsearch-ca.pem" ]
elasticsearch.ssl.verificationMode: "none"

To disregard the validity of SSL certificates, change this setting's value to 'none'.

#elasticsearch.ssl.verificationMode: full

=================== System: Logging ===================

Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'info'

#logging.root.level: debug

Enables you to specify a file where Kibana stores log output.

logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file

layout:

type: json

Logs queries sent to Elasticsearch.

#logging.loggers:

- name: elasticsearch.query

level: debug

Logs http responses.

#logging.loggers:

- name: http.server.response

level: debug

Logs system usage information.

#logging.loggers:

- name: metrics.ops

level: debug

=================== System: Other ===================

The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data

#path.data: data

Specifies the path where Kibana creates the process ID file.

pid.file: /run/kibana/kibana.pid

=================== Frequently used (Optional)===================

This section was automatically generated during setup.

#elasticsearch.ssl.certificateAuthorities: /etc/kibana/elasticsearch-ca.pem
#elasticsearch.ssl.certificateAuthorities: /var/lib/kibana/ca_1669211938854.crt
#elasticsearch.ssl.verificationMode: "none"
#elasticsearch.ssl.certificateAuthorities: /etc/kibana/fullchain1.pem
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: false, is_default_monitoring: false, type: elasticsearch, hosts: ['https://localhost:9200'], ca_trusted_fingerprint: cdacd71267e1bd13d8a4b87a00e44cf9cabd1}]
#allow_restricted_indices: true

xpack.encryptedSavedObjects.encryptionKey: 98999e42c9fa4cc219c63df6bb0
xpack.reporting.encryptionKey: 31037672066b6e16551c43e1a4d
xpack.security.encryptionKey: b819fcf7398730825f66832a2b
#xpack.reporting.kibanaServer.hostname: localhost
logging.appenders.file.type: file
logging.appenders.file.fileName: /var/log/kibana/kibana.log
logging.appenders.file.layout.type: json
logging.root.appenders: [default, file]
xpack.fleet.agents.enabled: true >"

log
log1902×600 252 KB

2

Hi @Anass.EL,

Welcome to the community. It looks like Kibana and Elasticsearch are struggling to connect. Can you confirm that you have followed the steps in this topic? Can you also share the log as text or code rather than an image to make it easier to read?

3

Hello @carly.richmond,

Thank you for your reply,

When using curl -XGET ip:9200/
i get the error: curl: (52) Empty reply from server

the kibana.log:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:13.029+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":96862},"trace":{"id":"39953ba6190012e4e03b43a7bd6a156b"},"transaction":{"id":"009cff7f04ee8d4b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.484+00:00","message":"Plugin \"cloudChat\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":96862},"trace":{"id":"39953ba6190012e4e03b43a7bd6a156b"},"transaction":{"id":"009cff7f04ee8d4b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.484+00:00","message":"Plugin \"cloudExperiments\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":96862},"trace":{"id":"39953ba6190012e4e03b43a7bd6a156b"},"transaction":{"id":"009cff7f04ee8d4b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.484+00:00","message":"Plugin \"cloudFullStory\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":96862},"trace":{"id":"39953ba6190012e4e03b43a7bd6a156b"},"transaction":{"id":"009cff7f04ee8d4b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.484+00:00","message":"Plugin \"cloudGainsight\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":96862},"trace":{"id":"39953ba6190012e4e03b43a7bd6a156b"},"transaction":{"id":"009cff7f04ee8d4b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.492+00:00","message":"Plugin \"profiling\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":96862},"trace":{"id":"39953ba6190012e4e03b43a7bd6a156b"},"transaction":{"id":"009cff7f04ee8d4b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.574+00:00","message":"http server running at http://0.0.0.0:5601","log":{"level":"INFO","logger":"http.server.Preboot"},"process":{"pid":96862},"trace":{"id":"39953ba6190012e4e03b43a7bd6a156b"},"transaction":{"id":"009cff7f04ee8d4b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.630+00:00","message":"Setting up [1] plugins: [interactiveSetup]","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":96862},"trace":{"id":"39953ba6190012e4e03b43a7bd6a156b"},"transaction":{"id":"009cff7f04ee8d4b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.683+00:00","message":"The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set \"xpack.reporting.roles.enabled\" to \"false\" to adopt the future behavior before upgrading.","log":{"level":"WARN","logger":"config.deprecation"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.941+00:00","message":"Setting up [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.953+00:00","message":"CustomBrandingService registering plugin: customBranding","log":{"level":"INFO","logger":"custom-branding-service"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:26.957+00:00","message":"TaskManager is identified by the Kibana UUID: 93dee05b-ff23-4a28-9cbe-1aa3e4f7db90","log":{"level":"INFO","logger":"plugins.taskManager"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:27.030+00:00","message":"Session cookies will be transmitted over insecure connections. This is not recommended.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:27.053+00:00","message":"Session cookies will be transmitted over insecure connections. This is not recommended.","log":{"level":"WARN","logger":"plugins.security.config"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:27.062+00:00","message":"Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: X8f5WYxZ5Ge9CTj7/dKl0TQ+ZK7eKpEJLz5csRszDLk=","log":{"level":"INFO","logger":"plugins.encryptedSavedObjects"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:27.077+00:00","message":"Email Service Error: Email connector not specified.","log":{"level":"INFO","logger":"plugins.notifications"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:27.185+00:00","message":"Found 'server.host: \"0.0.0.0\"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.","log":{"level":"WARN","logger":"plugins.reporting.config"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:27.207+00:00","message":"Installing common resources shared between all indices","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:27.458+00:00","message":"Registered task successfully [Task: cloud_security_posture-stats_task]","log":{"level":"INFO","logger":"plugins.cloudSecurityPosture"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:27.771+00:00","message":"Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 22.04 OS. Automatically enabling Chromium sandbox.","log":{"level":"INFO","logger":"plugins.screenshotting.config"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:16:28.206+00:00","message":"Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:18:27.810+00:00","message":"**Unable to retrieve version information from Elasticsearch nodes. Request timed out","log**":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:36:27.208+00:00","message":"Timeout: it took more than 1200000ms","error":{"message":"Timeout: it took more than 1200000ms","type":"Error","stack_trace":"Error: Timeout: it took more than 1200000ms\n    at Timeout._onTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:49:20)\n    at listOnTimeout (node:internal/timers:559:17)\n    at processTimers (node:internal/timers:502:7)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:36:27.210+00:00","message":"Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms","error":{"message":"Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms","type":"Error","stack_trace":"Error: Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms\n    at ResourceInstaller.installWithTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:62:13)\n    at ResourceInstaller.installCommonResources (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:76:5)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":96862},"trace":{"id":"da6f1543ca7adfc987bc34b1e64e4d8b"},"transaction":{"id":"6c69cecec9dd149b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:40:18.911+00:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":96862}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:40:18.912+00:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":96862}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:40:18.912+00:00","message":"Monitoring stats collection is stopped","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":96862}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-06-02T10:40:18.929+00:00","message":"no elements in sequence","error":{"message":"no elements in sequence","type":"EmptyError","stack_trace":"Error: \n    at _super (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/createErrorClass.js:7:26)\n    at new EmptyErrorImpl (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/EmptyError.js:6:5)\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/first.js:13:263\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/throwIfEmpty.js:14:86\n    at OperatorSubscriber._this._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:56:21)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at OperatorSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at OperatorSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:104:39\n    at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)\n    at ReplaySubject.Subject.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:98:24)\n    at Object.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/share.js:80:30)\n    at ConsumerObserver.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:138:33)\n    at SafeSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at SafeSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/takeUntil.js:10:141\n    at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)\n    at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:69:34\n    at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)\n    at Subject.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:59:24)\n    at ElasticsearchService.stop (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-server-internal/src/elasticsearch_service.js:127:16)\n    at Server.stop (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/server.js:380:30)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at Root.shutdown (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/root/index.js:77:5)"},"log":{"level":"FATAL","logger":"root"},"process":{"pid":96862},"trace":{"id":"8e683622d2fe25983d221314f723c5bb"},"transaction":{"id":"497088578edf5fba"}}

the elasticsearch.log

[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.apache.logging.log4j.core@8.5.2/org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.apache.logging.log4j.core@8.5.2/org.apache.logging.log4j.core.Logger.log(Logger.java:161)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2017)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1983)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:750)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.elasticsearch.server@8.5.2/org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler.onNonFatalUncaught(ElasticsearchUncaughtExceptionHandler.java:60)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at org.elasticsearch.server@8.5.2/org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler.uncaughtException(ElasticsearchUncaughtExceptionHandler.java:45)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:701)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:696)
[2022-11-24T14:06:16,489][WARN ][stderr                   ] [ip-172-31-46-9] 	at java.base/java.lang.Thread.dispatchUncaughtException(Thread.java:3007)
[2022-11-24T14:06:16,380][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ip-172-31-46-9] uncaught exception in thread [process reaper (pid 2131)]
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "modifyThread")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at org.elasticsearch.secure_sm.SecureSM.checkThreadAccess(SecureSM.java:166) ~[?:?]
	at org.elasticsearch.secure_sm.SecureSM.checkAccess(SecureSM.java:120) ~[?:?]
	at java.lang.Thread.checkAccess(Thread.java:2360) ~[?:?]
	at java.lang.Thread.setDaemon(Thread.java:2308) ~[?:?]
	at java.lang.ProcessHandleImpl.lambda$static$0(ProcessHandleImpl.java:103) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:637) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:928) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1021) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
	at jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186) ~[?:?]
[2022-11-24T14:06:16,637][INFO ][o.e.n.Node               ] [ip-172-31-46-9] stopped
[2022-11-24T14:06:16,638][INFO ][o.e.n.Node               ] [ip-172-31-46-9] closing ...
[2022-11-24T14:06:16,645][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [ip-172-31-46-9] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-15829101673581128703/geoip-databases/eyqfHHD1TCuot9iERLroDg/GeoLite2-Country.mmdb]
[2022-11-24T14:06:16,645][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [ip-172-31-46-9] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-15829101673581128703/geoip-databases/eyqfHHD1TCuot9iERLroDg/GeoLite2-ASN.mmdb]
[2022-11-24T14:06:16,645][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [ip-172-31-46-9] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-15829101673581128703/geoip-databases/eyqfHHD1TCuot9iERLroDg/GeoLite2-City.mmdb]
[2022-11-24T14:06:16,650][INFO ][o.e.n.Node               ] [ip-172-31-46-9] closed
4

Thanks for sharing @Anass.EL. I can see two potential problems:

Do you have SSL or the security settings disabled or misconfigured? As of 8.x security settings are enabled by default and we don't recommend switching them off.

It looks like you have misconfigured security settings. Have you followed the steps to register Kibana with the enrolment token?

Hope that helps.

5

Thanks for your reply @carly.richmond,
the security settings are not disabled and i have SSL
normally the enrollement was done at the start and everything worked just fine,

and now i tried to re-generate the enrollement token but i get the error: ERROR: Failed to determine the health of the cluster. Unexpected http status [403]

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.