Can I upgrade my existing ELK setup to 8.x?

Elastic Stack Elasticsearch
1

Hello,

I have a query regarding the ELK stack upgrade from 7.17.15 to 8.11 or higher?

Please note the current set up is of a production server and is on 7.17.15 and is also basic version (NOT a licensed version) also this is a stand alone server.

# curl -X GET "localhost:9200/"
{
  "name" : "ip-x-x-x-x",
  "cluster_name" : "abc",
  "cluster_uuid" : "xyz",
  "version" : {
    "number" : "7.17.15",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "0b8ecfb4378335f4689c4223d1f1115f16bef3ba",
    "build_date" : "2023-11-10T22:03:46.987399016Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"

filebeat version

filebeat version 7.17.15 (arm64), libbeat 7.17.15 [b474d2803ed2961f23f614d7213d9099fb0b4354 built 2023-11-08 19:08:34 +0000 UTC]
root@ip-192-168-135-26:~#

The purpose of the upgrading is because we are currently facing issues which filebeat indices hitting the main ELK server from the different client servers which are on filebeat 8.x

# curl -X GET "localhost:9200/_cat/indices/filebeat-8*?v"
health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat-8.11.1            neEszcRuQ3WFEdk_lZw7Ng   1   1  339526435            0    152.2gb        152.2gb
yellow open   filebeat-8.11.2            ytS9uDDvQDSeoLCtinUNFQ   1   1   33462501            0     10.6gb         10.6gb
yellow open   filebeat-8.10.4            HbQfL0pGS_ClJLLBWGFYUQ   1   1    7184754            0      1.5gb          1.5gb
yellow open   filebeat-8.10.4-2024.01.12 7ah5nAK6ShSf8oQhCSar3w   1   1      51352            0     10.2mb         10.2mb
yellow open   filebeat-8.11.2-2024.01.12 ATocymT0SVy1sYqYmivaxg   1   1    1224750            0    405.2mb        405.2mb
yellow open   filebeat-8.11.2-2024.01.11 4CmGRMeURxGMyatRaPf-Wg   1   1    1763247            0    578.6mb        578.6mb

Thanks,
Ravi

2

Hello,

Please can I know if I can upgrade the ELK stack from 7.17 to 8.11 or higher?

We have ILM implemented on this production server and xpack.security.enabled is also false.

What things we need to consider before making the move on the upgrade plan and is there any procedure to follow?

Thanks,
Ravi

3

Yes. Just start here: Upgrade Elasticsearch | Elasticsearch Guide [8.12] | Elastic

4

Hello,

I have tried running this on one of the test instance. The ELK stack version over there is on 7.17.13 and is a Basic version.

# curl -X GET "localhost:9200/"
{
  "name" : "xyz",
  "cluster_name" : "abc",
  "cluster_uuid" : "lmn",
  "version" : {
    "number" : "7.17.13",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "2b211dbb8bfdecaf7f5b44d356bdfe54b1050c13",
    "build_date" : "2023-08-31T17:33:19.958690787Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Had to resolve the issues for the depreciated settings.

  1. Critical
    Security features are enabled by default for all licenses in versions 8.0 and later
    In Elasticsearch 8.0 the [xpack.security.enabled] setting will always default to true. In your environment, the value is not set and you need to set it before upgrading, along with other required security settings. See Migrating to 7.14 | Elasticsearch Guide [7.16] | Elastic to secure your cluster. To explicitly disable security, set [xpack.security.enabled] to "false" (not recommended). (nodes impacted: [ELK-test])

  2. Warning
    Disk watermarks do not treat single-node clusters differently in versions 8.0 and later.
    Disk watermarks do not treat single-node clusters differently in versions 8.0 and later, which will affect the behavior of this cluster. Set [cluster.routing.allocation.disk.watermark.enable_for_single_data_node] to [true] to adopt the future behavior before upgrading. (nodes impacted: [ELK-test, ELK-test])

Faced with issue #1 on the test environment while fixing the depreciation issues.

---------------------------------- Security ----------------------------------

Enable security features

#xpack.security.enabled: true

  1. With commenting out the field, "xpack.security.enabled" in the elasticsearch.yml file and after restarting the elasticsearch.service

  2. When I tried to access the Kibana GUI, I have received the 503 error as per below and because of which the Kibana was not showing up.

{"statusCode":503,"error":"Service Unavailable", "message": License in not available."}

Question: Do we need the licensing to enable to xpack.security feature?

Thanks,
Ravi

5

No Security is a basic Feature.

There are some commercial features like SAML/SSO

But authentication/ authorization and SSL are all basic features.

That is probably due to Kibana not starting correctly and not being able to connect to elasticsearch. That license message is a symptom of that. It's not really a license problem. It's Kibana not being able to connect to elasticsearch most of the time.

You can share your elasticsearch.yml and kibana.yml and we might be able to help.

Please format your code with 3 back ticks before and after the code..

6

Hello,

Please find the details below on the elasticsearch.yml and kibana.yml

# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: localhost 
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
# single node elastic stack
discovery.type: single-node
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Security ----------------------------------
# Enable security features
#xpack.security.enabled: true

#
#                                 *** WARNING ***
#
# Elasticsearch security features are not enabled by default.
# These features are free, but require configuration changes to enable them.
# This means that users don’t have to provide credentials and can get full access
# to the cluster. Network connections are also not encrypted.
#
# To protect your data, we strongly encourage you to enable the Elasticsearch security features. 
# Refer to the following documentation for instructions.


# https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html

kibana.yml

# cat /etc/kibana/kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://127.0.0.1:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Kibana can also authenticate to Elasticsearch via "service account tokens".
# If may use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid

# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

Thanks,
Ravi

7

In your kibana.yml

That means kibana can only be accessed from the server kibana is running on is that what you want?

and to verify when you run

curl http://localhost:9200 you get a valid response

localhost and 127.0.0.1 are not always equal perhaps try localhost

8

Hello,

The existing setup is a single node solution. Also, I have tried to change the configs in kibana.yml as per the below.

server.port: 5601

server.host: "localhost"

elasticsearch.hosts: ["localhost:9200"]

I have restarted the kibana service after the changes per above. But now the kibana service is not coming up and the journalctl for the same displays as below.

Mar 03 19:24:55 ELK-test systemd[1]: kibana.service: Scheduled restart job, restart counter is at 1.
Mar 03 19:24:55 ELK-test systemd[1]: Stopped Kibana.
Mar 03 19:24:55 ELK-test systemd[1]: kibana.service: Consumed 12.092s CPU time.
Mar 03 19:24:55 ELK-test systemd[1]: Started Kibana.
Mar 03 19:24:55 ELK-test kibana[2136]: Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disa>
Mar 03 19:25:05 ELK-test kibana[2136]:  FATAL  Error: [config validation of [elasticsearch].hosts]: types that failed validation:
Mar 03 19:25:05 ELK-test kibana[2136]: - [config validation of [elasticsearch].hosts.0]: expected value of type [string] but got [Array].
Mar 03 19:25:05 ELK-test kibana[2136]: - [config validation of [elasticsearch].hosts.1.0]: expected URI with scheme [http|https].
Mar 03 19:25:05 ELK-test systemd[1]: kibana.service: Main process exited, code=exited, status=78/CONFIG
Mar 03 19:25:05 ELK-test systemd[1]: kibana.service: Failed with result 'exit-code'.
Mar 03 19:25:05 ELK-test systemd[1]: kibana.service: Consumed 12.018s CPU time.
Mar 03 19:25:08 ELK-test systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Mar 03 19:25:08 ELK-test systemd[1]: Stopped Kibana.
Mar 03 19:25:08 ELK-test systemd[1]: kibana.service: Consumed 12.018s CPU time.
Mar 03 19:25:08 ELK-test systemd[1]: Started Kibana.
Mar 03 19:25:09 ELK-test kibana[2161]: Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disa>
Mar 03 19:25:19 ELK-test kibana[2161]:  FATAL  Error: [config validation of [elasticsearch].hosts]: types that failed validation:
Mar 03 19:25:19 ELK-test kibana[2161]: - [config validation of [elasticsearch].hosts.0]: expected value of type [string] but got [Array].
Mar 03 19:25:19 ELK-test kibana[2161]: - [config validation of [elasticsearch].hosts.1.0]: expected URI with scheme [http|https].
Mar 03 19:25:19 ELK-test systemd[1]: kibana.service: Main process exited, code=exited, status=78/CONFIG
Mar 03 19:25:19 ELK-test systemd[1]: kibana.service: Failed with result 'exit-code'.
Mar 03 19:25:19 ELK-test systemd[1]: kibana.service: Consumed 12.060s CPU time.
Mar 03 19:25:22 ELK-test systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Mar 03 19:25:22 ELK-test systemd[1]: Stopped Kibana.
Mar 03 19:25:22 ELK-test systemd[1]: kibana.service: Consumed 12.060s CPU time.
Mar 03 19:25:22 ELK-test systemd[1]: kibana.service: Start request repeated too quickly.
Mar 03 19:25:22 ELK-test systemd[1]: kibana.service: Failed with result 'exit-code'.
Mar 03 19:25:22 ELK-test systemd[1]: Failed to start Kibana.

Thanks,
Ravi

9

Per the error..

not correct

elasticsearch.hosts: ["http://localhost:9200"]

need the http

10

Hello,

I am sorry that I did a mistake in the configs for kibana.yml and thanks for guiding me correcting on the same.

Currently kibana service is up and I will set the xpack security to true and see what it says.

Thank you again.

Thanks,
Ravi

1 Like
11

Hello,

After the above changes I am able to see that the kibana service is starting normally but when I accessed the GUI and there I am facing error as, "Kibana server is not ready yet".

And the logs says as per the below.

kibana.log

{"type":"log","@timestamp":"2024-03-04T12:15:07+00:00","tags":["warning","process"],"pid":395,"message":"Error [ProductNotSupportedSecurityError]: The client is unable to verify that the server is Elasticsearch due to security privileges on the server side. Some functionality may not be compatible if the server is running an unsupported product.\n    at /usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:576:19\n    at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:369:9)\n    at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:291:11)\n    at IncomingMessage.emit (node:events:526:35)\n    at endReadableNT (node:internal/streams/readable:1359:12)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"}
{"type":"log","@timestamp":"2024-03-04T12:15:07+00:00","tags":["error","elasticsearch-service"],"pid":395,"message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}
"/var/log/kibana/kibana.log" 57007L, 45848484B

elasticsearch.log

[2024-03-04T12:22:29,118][INFO ][o.e.x.s.a.RealmsAuthenticator] [ELK-test] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2024-03-04T12:22:31,618][INFO ][o.e.x.s.a.RealmsAuthenticator] [ELK-test] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2024-03-04T12:22:34,118][INFO ][o.e.x.s.a.RealmsAuthenticator] [ELK-test] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2024-03-04T12:22:36,618][INFO ][o.e.x.s.a.RealmsAuthenticator] [ELK-test] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]

Thanks,
Ravi

12

Can you curl elasticsearch without credentials?

curl http://localhost:9200

Question you had security and abled and how you have a disabled?

From the error message kibana is trying to log into elasticsearch with the kibana_system

Did you reenable security?

Did you make updates to elasticsearch.yml and kibana.yml from what you showed above

13

Comments: I have comment out the following from /etc/kibana/kibana.yml

elasticsearch.username: "kibana_system"
elasticsearch.password: "pass"

Thanks,
Ravi

14

Right but as you can see from the curl response... Elastic is still expecting authentication...

So you will need to enable the

elasticsearch.username: "kibana_system"
elasticsearch.password: "pass"

It is unclear to me what state the security is in...

You did not answer did you have security enabled ... then you disabled?

also try

curl -u elastic http://localhost:9200

Authentication appears to is still active

15 
root@ELK-test:~# curl -u elastic http://localhost:9200
Enter host password for user 'elastic':

In the above it is prompting for the password.

Thanks,
Ravi

16

Yes...

So yes Elasticsearch will require authentication with that setting on... which is fine

Which means the you will also need to add this back in kibana.yml

elasticsearch.username: "kibana_system"
elasticsearch.password: "pass"
17

Hello,

I guess this was a missing bit in the elasticsearch.yml but the above entry in the kibana.yml already exists.

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana_system"
elasticsearch.password: "pass"

And I have tried with the below steps.

  1. root@ELK-test:/usr/share/elasticsearch/bin# ./elasticsearch-keystore list
    keystore.seed

  2. root@ELK-test:/usr/share/elasticsearch/bin# ./elasticsearch-keystore add -x "bootstrap.password"
    pass

  3. vim /etc/elasticsearch/elasticsearch.yml

---------------------------------- Security ----------------------------------

Enable security features

xpack.security.enabled: true
xpack.security.authc.realms.native.native1.order: 0

  1. systemctl restart elasticsearch.service

  2. root@ELK-test:~# systemctl status elasticsearch.service
    ● elasticsearch.service - Elasticsearch
    Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
    Active: active (running) since Mon 2024-03-04 17:59:39 UTC; 9s ago

But still the same error while accessing the GUI i.e. Kibana server is not ready yet.

And the logs stating as below.

elasticsearch.log

[2024-03-04T18:01:42,793][INFO ][o.e.x.s.a.RealmsAuthenticator] [ELK-test] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]

kibana.log

{"type":"log","@timestamp":"2024-03-04T17:59:40+00:00","tags":["error","elasticsearch-service"],"pid":387,"message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}

Thanks,
Ravi

18

@Ravi_Pattar Please slow down and answer the questions I am asking...

Notsure why you have

xpack.security.authc.realms.native.native1.order: 0

and

These are not correct....

do you have the elastic user password yes or no?

Did the curl this curl work ... yes or no?

curl -u elastic:password http://localhost:9200

Please show the results...

if so then we can move to the next step of fixing kibana...

from elasticsearch

./bin/elasticsearch-reset-password -u kibana_system

19 
root@ELK-test:~# curl -u elastic:password http://localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

Thanks,
Ravi

20

You need to put in the correct password for the curl

If you do not have it try

./bin/elasticsearch-reset-password -u elastic

Then try the curl with the new password