Changing elasticsearch/kibana IP address

Elastic Stack Kibana
1

Hello everyone

I've been searching and trying every manual on Internet for this issue, but at end, couldn't resolve the problem.

I installed elasticsearch/kibana via the manual here from debian packages.

guide/en/elasticsearch/reference/current/deb.html
guide/en/kibana/current/deb.html

Installing via this manual, generates and configures CA, and http , transport certificates for elastic and kibana.
Elasticsearch and kibana work fine.
server IP is:
network.host: 192.168.1.112

now I must move my server to another location and change the IP address of the server that IP will be: 192.168.200.112

by changing network.host in elasticsearch.yml , and restarting elasticsearch service, the curl to
https: // 192.168.200.112:9200
works fine.

but Kibana doesn't work, it needs to generate new Token.
and for generating new token, I think I must generate new certificates as you said here:
guide/en/elasticsearch/reference/8.7/update-node-certs.html

but this manual doesn't generate both HTTP and Transport keys and only i can generate transport key.
and by replacing it, when I want to create new token for kibana, I receive error:

ERROR: Failed to determine the health of the cluster. , with exit code 69

Please make a from scratch Document for actions must be taken for changing IP address of Elastic/kibana server and regenerating those certificates and keys it generates at installation and configs make automatically in kibana.yml

2

Hi @maissam_asgari,

Welcome! Which version of Elasticsearch are you trying to install? Are you trying to perform a clean install in the new host?

I would have thought the easiest thing to do is install Elasticsearch and Kibana on a new host following the install instructions and then following the steps for migrating your data.

Let us know if those steps help.

3

Thank you for reply

I have the latest version of elasticsearch, Installed already. Xpack security enabled, CA generated, hhtp and transport placed the way it do itself at installation. Just need to change the IP address of the machine and re-do all these security settings it did at installation itself, by me again.

Yeah, I thought about that, but these two networks don't have a connection and I would like to just transfer this first elastic machine to the second location and just change the IP.

4

Thanks for confirming. I assume you're not installing Elasticsearch in an airgapped environment? If so I would recommend checking out the airgapped documentation as that could impact some features.

If you're looking to regenerate certificates I would recommend having a look at elasticsearch-certutil if you haven't already.

Hope that helps!

5

I tried that, no success

It seems the best solution is to switch to Docker mode installation of Elasticsearch/kibana.

but for later updates, please add some instruction for easy changing IP address of elastic/kibana machine.

6

Hi @maissam_asgari welcome to the community and sorry you are having some issues.

What exactly did you try?

Following these instructions

If you regenerate the certs from scratch (and correctly) , then make sure the config all point to the new certs it will absolutely work.

What is often see is the HTTP certificate is not correctly generated.

That said it is not a super common task.

1 Like
8

Hello, Thank you for your reply

Let's simulate the situation in a clean environment

I have a Ubuntu Desktop 22.04 with IP address: 192.168.17.180

I installed elasticsearch from elasticsearch-8.14.0-amd64.deb I downloaded few days ago. based on the instruction Here

sudo dpkg -i elasticsearch-8.14.0-amd64.deb
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Curl: https://192.168.17.180:9200

{
  "name" : "ubuntulvm",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "cYRpPxFGQ4mS_YOqNN2y5A",
  "version" : {
    "number" : "8.14.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "8d96bbe3bf5fed931f3119733895458eab75dca9",
    "build_date" : "2024-06-03T10:05:49.073003402Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

then edited elasticsearch.yml (every time I installed elasticsearch and before starting it, edited the elasticsearch.yml i got error blow:)

{"error":{"root_cause":[{"type":"status_exception","reason":"Cluster state has not been recovered yet, cannot write to the [null] index"}],"type":"authentication_processing_error","reason":"failed to promote the auto-configured elastic password hash","caused_by":{"type":"status_exception","reason":"Cluster state has not been recovered yet, cannot write to the [null] index"}},"status":503}

but when elasticsearch started one time and then I edited the yml file, it's ok

cluster.name: my-application
node.name: node-1
network.host: 192.168.17.180

then:
sudo systemctl restart elasticsearch

Curl: https://192.168.17.180:9200

{
  "name" : "node-1",
  "cluster_name" : "my-application",
  "cluster_uuid" : "cYRpPxFGQ4mS_YOqNN2y5A",
  "version" : {
    "number" : "8.14.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "8d96bbe3bf5fed931f3119733895458eab75dca9",
    "build_date" : "2024-06-03T10:05:49.073003402Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Now, Installing Kibana from kibana-8.14.0-amd64.deb based on instruction Here

sudo dpkg -i kibana-8.14.0-amd64.deb
sudo systemctl daemon-reload
sudo systemctl enable kibana.service

then edited kibana.yml

server.host: "0.0.0.0"

then:

sudo systemctl start kibana.service
sudo systemctl status kibana.service

it says: Go to http://0.0.0.0:5601/?code=829206 to get started.

based on instruction i generated elastic token:

bin/elasticsearch-create-enrollment-token -s kibana

at this state, Elasticsearch and kibana are up and working properly
after some time I need to move this PC or VM to another network that the IP address will change

Let's think I just moved the machin and changed the IP to (for example) 192.168.17.170
changed the elasticsearch.yml

network.host: 192.168.17.170

then

sudo systemctl start elasticsearch

curl: https://192.168.17.170:9200

{
  "name" : "node-1",
  "cluster_name" : "my-application",
  "cluster_uuid" : "cYRpPxFGQ4mS_YOqNN2y5A",
  "version" : {
    "number" : "8.14.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "8d96bbe3bf5fed931f3119733895458eab75dca9",
    "build_date" : "2024-06-03T10:05:49.073003402Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Elastic is OK

now let's start Kibana:
before that I commented the latest part of kibana.yml

#elasticsearch.hosts: ['https://192.168.17.180:9200']
#elasticsearch.serviceAccountToken: xxxxxxxxx
#elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1718709552933.crt]
#xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, ho>

then started kibana. web page brings the token configuration page:

 Go to http://0.0.0.0:5601/?code=077168 to get started.

so I try to generate new token:
at this state I got an error:

15:03:50.251 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [192.168.17.170]; the server provided a certificate with subject name [CN=ubuntulvm], fingerprint [87bb70919bd177646d9d397f37b3d2b8d9c8604c], no keyUsage and extendedKeyUsage [serverAuth]; the certificate is valid between [2024-06-18T11:02:15Z] and [2026-06-18T11:02:15Z] (current time is [2024-06-18T11:33:50.247651775Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [IP:192.168.17.180,DNS:ubuntulvm,IP:fe80:0:0:0:78b3:783b:b92e:14a3,IP:0:0:0:0:0:0:0:1,IP:127.0.0.1,DNS:localhost]; the certificate is issued by [CN=Elasticsearch security auto-configuration HTTP CA]; the certificate is signed by (subject [CN=Elasticsearch security auto-configuration HTTP CA] fingerprint [d90c5426c909818c197c7270ef00998ff74966b5] {trusted issuer}) which is self-issued; the [CN=Elasticsearch security auto-configuration HTTP CA] certificate is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: Composite-Trust{JDK-trusted-certs,StoreTrustConfig{path=certs/http.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX}})])
java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.17.170 found

so now I use the Link you gave:

the part: Basic security (Elasticsearch + Kibana)

first, generatde new CA with default name (elastic-stack-ca.p12) and no password

./bin/elasticsearch-certutil ca

previous configuration of elasticsearch.yml is:

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["ubuntulvm"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

so I generate the transport key with the same name "transport.p12"

/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
Please enter the desired output file [elastic-certificates.p12]: transport.p12
with no password
sudo mv transport.p12 /etc/elasticsearch/certs/
sudo cp  elastic-stack-ca.p12 /etc/elasticsearch/certs/

chown and chmod both files to become like the previous ones

removing passwords from elasticsearch-keystore because I generated CA and transport without password

sudo ./bin/elasticsearch-keystore remove xpack.security.transport.ssl.truststore.secure_password
sudo ./bin/elasticsearch-keystore remove xpack.security.transport.ssl.keystore.secure_password

systemctl stop kibana.service
sudo systemctl restart elasticsearch.service

the Elastic doesn't start!

Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xeu elasticsearch.service" for details.

reviewing the log file it says:

failed to load SSL configuration [xpack.security.http.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/http.p12] because the file does not exist

because I moved the old certs folder to certs-old and mkdir certs and placed elastic-stack-ca.p12 and transport.p12 inside it

so based on the part two of the link you provided: Basic security plus secured HTTPS traffic (Elastic Stack)

 sudo ./bin/elasticsearch-certutil http
Generate a CSR? [y/N]N
Use an existing CA? [y/N]y
CA Path: /etc/elasticsearch/certs/elastic-stack-ca.p12
no password
5y
Generate a certificate per node? [y/N]N (I have just one node)

Enter all the hostnames that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

node-1

Enter all the IP addresses that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

192.168.17.170

no password for "http.p12"

What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]

then

sudo mv elasticsearch-ssl-http.zip /etc/elasticsearch/certs/
cd /etc/elasticsearch/certs
unzip elasticsearch-ssl-http.zip
cd elasticsearch
mv http.p12 ../
chmod 660 http.p12

also: (because I didn't give password to http.p12
sudo /usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.http.ssl.keystore.secure_password

now starting elasticsearc.service
It started properly and working

so let's start kibana and generate enrollment token
Kibana started, it says go to http ://0.0.0.0:5601/?code=888567

generating enrollment token:

bin/elasticsearch-create-enrollment-token -s kibana

Unable to create enrollment token for scope [kibana]

ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration Keystore doesn't contain any PrivateKey entries where the associated certificate is a CA certificate, with exit code 73

I don't know what to do after this,
with the link you gave, I could generate CA, transport and http and start elasticsearch, before that I had problem there too.

Please help me complete this process.

Thank you

1 Like
9

@maissam_asgari Thank you for such as detailed response and looks like you are doing pretty much everything right.... except for 1 last thing, you are so close.

It Turns out once you set / create / manually configure security... you can no longer use the token enrollment process to enroll Kibana .... I wrote this up here.

Ugggh yes sorry about that :frowning:

So all you need to do is manually configure kibana with the following

elasticsearch.hosts: ["https://<elastic_ip_or_host>:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "pass"

# The CA you created the elasticsearch HTTP cert with, copy to where you want and reference it
elasticsearch.ssl.certificateAuthorities: /etc/kibana/elasticsearch-ca.pem

I think the basic directions are here

If you did not setup the kibana_system user account you can set it up with the command here

Let us know if it works...

1 Like
10

Hello again

From the previous state, the http cert for kibana was created here:

so I created a dir /etc/kibana/config/
and placed elasticsearch-ca.pem inside it
added and un-commented these lines inside kibana.yml

elasticsearch.hosts: ["https://192.168.17.170:9200"]

elasticsearch.username: "kibana_system"
elasticsearch.password: "xxxxxxxxxxxxxxx"

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/config/elasticsearch-ca.pem" ]

for the kibana_system password:

sudo ./bin/elasticsearch-setup-passwords auto

Failed to authenticate user 'elastic' against https://192.168.17.170:9200/_security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /etc/elasticsearch/elasticsearch.keystore

You can use the `elasticsearch-reset-password` CLI tool to reset the password of the 'elastic' user


ERROR: Failed to verify bootstrap password, with exit code 78

so tried this:

sudo ./bin/elasticsearch-reset-password -u kibana_system

This tool will reset the password of the [kibana_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y


Password for the [kibana_system] user successfully reset.
New value: xxxxxxxxxxxxxxxx

then

sudo systemctl restart kibana

sudo systemctl status kibana

● kibana.service - Kibana
     Loaded: loaded (/lib/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2024-06-19 17:30:33 +0330; 646ms ago
       Docs: https://www.elastic.co
    Process: 4319 ExecStart=/usr/share/kibana/bin/kibana (code=exited, status=1/FAILURE)
   Main PID: 4319 (code=exited, status=1/FAILURE)
        CPU: 10.335s

inside the kibana.log

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-06-19T17:30:33.685+03:30","message":"Reason: EACCES: permission denied, open '/etc/kibana/config/elasticsearch-ca.pem'\nError: EACCES: permission denied, open '/etc/kibana/config/elasticsearch-ca.pem'\n

so changed owner of the file (before changing, it was root:elasticsearch, maybe because I unziped and copied it from elastic folder)

sudo chown root:kibana elasticsearch-ca.pem

sudo systemctl start kibana

Now It's Running and Working :slight_smile:

I'm glad it's documented with all details for others may have this problem.

Thank you so much for your kind responses.

1 Like