Kibana : Internal Server Error

Elastic Stack Kibana
1

Hi,

I am running Kibana 7.11 and I am getting this Internal Server Error when I try to get logs on the Kibana Portal.


 log   [11:09:04.725] [info][listening] Server running at http://0.0.0.0:5601
  log   [11:09:05.303] [info][server][Kibana][http] http server running at http://0.0.0.0:5601
  log   [11:09:05.310] [info][plugins][watcher] Your basic license does not support watcher. Please upgrade your license.
  log   [11:09:05.314] [info][crossClusterReplication][plugins] Your basic license does not support crossClusterReplication. Please upgrade your license.
  log   [11:09:05.314] [info][kibana-monitoring][monitoring][monitoring][plugins] Starting monitoring stats collection
  log   [11:09:06.055] [warning][plugins][reporting] Enabling the Chromium sandbox provides an additional layer of protection.
 error  [11:09:32.279]  Error: Internal Server Error
    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:121:19)
    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:75:19)
    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:70:17)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:45:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:312:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:221:9)

Please recommend if anyone has an idea of how to fix this issue?

Thank you.

2

Hi @sanju_timsina Welcome to the community!
Your Kibana logs point to an issue with a http request that errored out but we can't make out the exact request that threw the error.
The logs prior to the error show that Kibana appears to be up and running (log line 1 and 2).

The log messages from the watcher, crossClusterReplication aren't anything to be concerned about, they're only warning you that the basic license doesn't support those features.

Do you have elasticsearch running? If Kibana can't access elasticsearch, we'll get an error.

If you do, then there's a different problem but we need more verbose logs that what you have (use the --verbose logging option).

The logs should then give a bit more information as to what's causing the issue.
Tina

3

Hi Tina,

Yes, the elasticsearch is running. I get this error when I try to access the logs on the Kibana.

Please find the logs below.

  log   [15:02:36.508] [debug][server][Kibana][http] registering route handler for [/]
  log   [15:02:36.509] [debug][server][Kibana][http] registering route handler for [/core]
  log   [15:02:36.511] [debug][server][Kibana][http] registering route handler for [/status]
  log   [15:02:36.515] [info][server][Kibana][http] http server running at http://0.0.0.0:5601
  log   [15:02:36.516] [debug][eventLog][plugins] initializing elasticsearch resources starting
  log   [15:02:36.516] [debug][eventLog][plugins] esContext: callEs(transport.request) calls: {"method":"GET","path":"/_ilm/policy/kibana-event-log-policy"}
  log   [15:02:36.526] [debug][licensing][plugins] Imported license information from Elasticsearch:type: basic | status: active | expiry date: Invalid date
  log   [15:02:36.527] [info][plugins][watcher] Your basic license does not support watcher. Please upgrade your license.
  log   [15:02:36.528] [debug][logging] Updating logging config for context [plugins.security]
  log   [15:02:36.530] [info][crossClusterReplication][plugins] Your basic license does not support crossClusterReplication. Please upgrade your license.
  log   [15:02:36.531] [info][kibana-monitoring][monitoring][monitoring][plugins] Starting monitoring stats collection
  log   [15:02:36.540] [debug][plugins][spaces] Checking for existing default space
  log   [15:02:36.552] [debug][eventLog][plugins] esContext: callEs(transport.request) result: {"kibana-event-log-policy":{"version":1,"modified_date":"2021-02-15T20:52:08.552Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}}}}}
  log   [15:02:36.552] [debug][eventLog][plugins] esContext: callEs(indices.existsTemplate) calls: {"name":".kibana-event-log-7.11.0-template"}
  log   [15:02:36.556] [debug][plugins][spaces] Default space already exists
  log   [15:02:36.557] [debug][status][plugins][spaces] available: ready
  log   [15:02:36.561] [debug][kibana-monitoring][monitoring][monitoring][plugins] [null] default admin email setting found, sending [kibana_settings] monitoring document.
  log   [15:02:36.562] [debug][kibana-monitoring][monitoring][monitoring][plugins] Uploading bulk stats payload to the local cluster
  log   [15:02:36.564] [debug][eventLog][plugins] esContext: callEs(indices.existsTemplate) result: true
  log   [15:02:36.564] [debug][eventLog][plugins] esContext: callEs(indices.existsAlias) calls: {"name":".kibana-event-log-7.11.0"}
  log   [15:02:36.567] [debug][eventLog][plugins] esContext: callEs(indices.existsAlias) result: false
  log   [15:02:36.567] [debug][eventLog][plugins] esContext: callEs(indices.create) calls: {"index":".kibana-event-log-7.11.0-000001","body":{"aliases":{".kibana-event-log-7.11.0":{"is_write_index":true}}}}
  log   [15:02:36.574] [debug][kibana-monitoring][monitoring][monitoring][plugins] Uploaded bulk stats payload to the local cluster
  log   [15:02:36.576] [debug][eventLog][plugins] esContext: callEs(indices.create) error: {"message":"[invalid_alias_name_exception] Invalid alias name [.kibana-event-log-7.11.0], an index exists with the same name as the alias, with { index_uuid=\"kyaeesBzT2KYlbDEazDp_Q\" & index=\".kibana-event-log-7.11.0\" }","statusCode":400}
  log   [15:02:36.576] [error][eventLog][plugins] error initializing elasticsearch resources: error creating initial index: [invalid_alias_name_exception] Invalid alias name [.kibana-event-log-7.11.0], an index exists with the same name as the alias, with { index_uuid="kyaeesBzT2KYlbDEazDp_Q" & index=".kibana-event-log-7.11.0" }
  log   [15:02:36.577] [debug][eventLog][plugins] readySignal.signal(false)
  log   [15:02:36.577] [error][eventLog][plugins] initialization failed, events will not be indexed
  log   [15:02:36.973] [debug][plugins][reporting] Browser executable: /usr/share/kibana/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell
  log   [15:02:36.973] [warning][plugins][reporting] Enabling the Chromium sandbox provides an additional layer of protection.
  log   [15:02:36.976] [debug][esqueue][plugins][queue-worker][reporting] kmxtgiwg0clj8d43fd0j6gah - Created worker for reporting jobs
  log   [15:02:36.976] [debug][plugins][reporting] Start complete
  log   [15:02:38.296] [debug][plugins][taskManager] Running task endpoint:user-artifact-packager "endpoint:user-artifact-packager:1.0.0"
  log   [15:02:38.299] [debug][0][0][endpoint:user-artifact-packager:1][plugins][securitySolution] User manifest not available yet.
  log   [15:02:38.431] [debug][plugins][taskManager] Latest Monitored Stats: {"id":"14f358a9-a8d1-4ae8-837a-115e5d7efaaa","timestamp":"2021-03-31T19:02:38.425Z","status":"OK","last_update":"2021-03-31T19:02:38.425Z","stats":{"configuration":{"timestamp":"2021-03-31T19:02:34.419Z","value":{"request_capacity":1000,"max_poll_inactivity_cycles":10,"monitored_aggregated_stats_refresh_rate":60000,"monitored_stats_running_average_window":50,"monitored_task_execution_thresholds":{"default":{"error_threshold":90,"warn_threshold":80},"custom":{}},"poll_interval":3000,"max_workers":10},"status":"OK"},"runtime":{"timestamp":"2021-03-31T19:02:38.425Z","value":{"polling":{"last_successful_poll":"2021-03-31T19:02:38.298Z","last_polling_delay":"2021-03-31T19:02:34.417Z","duration":{"p50":330,"p90":330,"p95":330,"p99":330},"claim_conflicts":{"p50":0,"p90":0,"p95":0,"p99":0},"claim_mismatches":{"p50":0,"p90":0,"p95":0,"p99":0},"result_frequency_percent_as_number":{"Failed":0,"NoAvailableWorkers":0,"NoTasksClaimed":0,"RanOutOfCapacity":0,"RunningAtCapacity":0,"PoolFilled":100}},"drift":{"p50":4037,"p90":4037,"p95":4037,"p99":4037},"load":{"p50":0,"p90":9.000000000000004,"p95":10,"p99":10},"execution":{"duration":{"endpoint:user-artifact-packager":{"p50":4,"p90":4,"p95":4,"p99":4}},"result_frequency_percent_as_number":{"endpoint:user-artifact-packager":{"Success":100,"RetryScheduled":0,"Failed":0,"status":"OK"}}}},"status":"OK"},"workload":{"timestamp":"2021-03-31T19:02:36.546Z","value":{"count":8,"task_types":{"actions_telemetry":{"count":1,"status":{"idle":1}},"alerting_health_check":{"count":1,"status":{"idle":1}},"alerting_telemetry":{"count":1,"status":{"idle":1}},"alerts_invalidate_api_keys":{"count":1,"status":{"idle":1}},"apm-telemetry-task":{"count":1,"status":{"idle":1}},"endpoint:user-artifact-packager":{"count":1,"status":{"idle":1}},"lens_telemetry":{"count":1,"status":{"idle":1}},"security:endpoint-diagnostics":{"count":1,"status":{"idle":1}}},"schedule":[["60s",1],["5m",2],["60m",1],["720m",1]],"overdue":1,"estimated_schedule_density":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"status":"OK"}}} 
  log   [15:02:58.450] [debug][metrics] Refreshing metrics
  ops   [15:03:01.520]  memory: 143.8MB uptime: 0:00:34 load: [0.10 0.08 0.20] delay: 0.212
  log   [15:03:01.975] [debug][plugins][taskManager] Latest Monitored Stats: {"id":"14f358a9-a8d1-4ae8-837a-115e5d7efaaa","timestamp":"2021-03-31T19:03:01.969Z","status":"OK","last_update":"2021-03-31T19:03:01.969Z","stats":{"configuration":{"timestamp":"2021-03-31T19:02:34.419Z","value":{"request_capacity":1000,"max_poll_inactivity_cycles":10,"monitored_aggregated_stats_refresh_rate":60000,"monitored_stats_running_average_window":50,"monitored_task_execution_thresholds":{"default":{"error_threshold":90,"warn_threshold":80},"custom":{}},"poll_interval":3000,"max_workers":10},"status":"OK"},"runtime":{"timestamp":"2021-03-31T19:03:01.969Z","value":{"polling":{"last_successful_poll":"2021-03-31T19:02:58.974Z","last_polling_delay":"2021-03-31T19:02:34.417Z","duration":{"p50":6,"p90":233.70000000000005,"p95":330,"p99":330},"claim_conflicts":{"p50":0,"p90":0,"p95":0,"p99":0},"claim_mismatches":{"p50":0,"p90":0,"p95":0,"p99":0},"result_frequency_percent_as_number":{"Failed":0,"NoAvailableWorkers":0,"NoTasksClaimed":88,"RanOutOfCapacity":0,"RunningAtCapacity":0,"PoolFilled":13}},"drift":{"p50":4037,"p90":4037,"p95":4037,"p99":4037},"load":{"p50":0,"p90":0,"p95":0,"p99":10},"execution":{"duration":{"endpoint:user-artifact-packager":{"p50":4,"p90":4,"p95":4,"p99":4}},"result_frequency_percent_as_number":{"endpoint:user-artifact-packager":{"Success":100,"RetryScheduled":0,"Failed":0,"status":"OK"}}}},"status":"OK"},"workload":{"timestamp":"2021-03-31T19:02:36.546Z","value":{"count":8,"task_types":{"actions_telemetry":{"count":1,"status":{"idle":1}},"alerting_health_check":{"count":1,"status":{"idle":1}},"alerting_telemetry":{"count":1,"status":{"idle":1}},"alerts_invalidate_api_keys":{"count":1,"status":{"idle":1}},"apm-telemetry-task":{"count":1,"status":{"idle":1}},"endpoint:user-artifact-packager":{"count":1,"status":{"idle":1}},"lens_telemetry":{"count":1,"status":{"idle":1}},"security:endpoint-diagnostics":{"count":1,"status":{"idle":1}}},"schedule":[["60s",1],["5m",2],["60m",1],["720m",1]],"overdue":1,"estimated_schedule_density":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"status":"OK"}}}
  log   [15:03:02.873] [debug][authentication][plugins][security] Current license does not support any security features, authentication is not needed.
  log   [15:03:02.880] [debug][server][Kibana][cookie-session-storage][http] Error: Unauthorized
respons [15:03:02.868]  POST /api/ui_counters/_report 200 571ms - 9.0B
  log   [15:03:03.450] [debug][metrics] Refreshing metrics
  log   [15:03:06.202] [debug][authentication][plugins][security] Current license does not support any security features, authentication is not needed.
  log   [15:03:06.204] [debug][server][Kibana][cookie-session-storage][http] Error: Unauthorized
  log   [15:03:06.236] [debug][licensing][plugins] Requesting Elasticsearch licensing API
 error  [15:03:06.200]  Error: Internal Server Error
    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:121:19)
    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:75:19)
    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:70:17)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:45:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:312:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:221:9)

Thank you,
Sanju

4

These lines give a bit more info:

log   [15:03:06.202] [debug][authentication][plugins][security] Current license does not support any security features, authentication is not needed.

  log   [15:03:06.204] [debug][server][Kibana][cookie-session-storage][http] Error: Unauthorized

  log   [15:03:06.236] [debug][licensing][plugins] Requesting Elasticsearch licensing API

It seems to be an auth issue. Could you double check that your elasticsearch credentials in your kibana.yml file are correct and that you have the right Kibana credentials in your elasticsearch.yml config?

If you're using the basic license, then security should be available. You might also want to make sure that security is enabled.

5

Credentials in elasticsearch and kibana seems to be correct.

Please take a look at below configuration file for elasticsearch and kibana.

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
discovery.type: single-node

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://0.0.0.0:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid

# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"
6

These are still commented out, you may want to uncomment them and set them to the auth details you have configured.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.