Hi all,
I'm running into an issue migrating machines in a cluster. Quick breakdown of the setup:
We had a cluster running in one site on one network subnet (network a). We are being ushered out of the building, so we bought some equipment to make up for the move. This included new master (head04..06) and data nodes for the ES cluster.
We stood up the swing equipment on the new network (network b) and were able to attach all of it to the existing cluster. We used the elastic-stack-ca.p12 file on head02 (a master node) which was used for ALL of the nodes in the cluster, new and old. We use the default .p12 keystore option.
ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --name <host> --dns <fqdn> --ip <ip-address>
We migrated data off a handful of data nodes, and prepped them as well as a couple masters to take to the new data center (including head02).
Once we moved the machines, I acquired new certs specifically using the elastic-stack-ca.p12 on head02 to replace the ones in the config dirs of the only the moved gear. head02 was not part of the cluster at this time. I was getting errors when I tried to start Elasticsearch on head02 (after changing the .p12 keystore) to rejoin the cluster:
[WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [<head04>]client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/<head04-IP>:9300, remoteAddress=/<head02-IPb>:34416, profile=default}
Current configuration on a master node in the cluster:
cluster.max_shards_per_node: 5000
cluster.name: <cluster>
discovery.seed_hosts:
- <head01-IPa>:9300
- <head01-IPb>:9300
- <head02-IPb>:9300
- <head03-IPb>:9300
- <head04-IPb>:9300
- <head05-IPb>:9300
- <head06-IPb>:9300
network.host: <host-IP>
node.data: 'false'
node.ingest: 'false'
node.master: 'true'
node.name: es-head04-master
path.data: "/var/lib/elasticsearch/es-head04-master"
path.logs: "/var/log/elasticsearch/es-head04-master"
path.repo:
- "/backup"
xpack.graph.enabled: 'false'
xpack.ml.enabled: 'false'
xpack.monitoring.collection.enabled: 'false'
xpack.security.enabled: 'true'
xpack.security.http.ssl.certificate: config/es-head04-http-cert.pem
xpack.security.http.ssl.enabled: 'true'
xpack.security.http.ssl.key: config/es-head04-http-key.pem
xpack.security.transport.ssl.enabled: 'true'
xpack.security.transport.ssl.keystore.path: config/es-head04.p12
xpack.security.transport.ssl.truststore.path: config/es-head04.p12
xpack.security.transport.ssl.verification_mode: full
xpack.watcher.enabled: 'false'
Each data and master node has a similiar xpack.secuirty.transport.ssl.keystore.path and truststore.path, except with their own .p12 made from the ca on head02 looking like:
xpack.security.transport.ssl.keystore.path: config/<host>.p12
xpack.security.transport.ssl.truststore.path: config/<host>.p12
I have also tried changing the xpack.security.transport.ssl.truststore.path to the elastic-stack-ca.p12 that I copied to all of the nodes in the cluster from head02, but then that proceeded to make nothing trust anything. What would be the best way to get machines that I've moved back into the cluster (and future machines we've still to move)? I cannot use their old certs as they are specifically tied to the old IP address of the machine. Should I make a new CA and issue brand new certs for all of the nodes in the cluster? And how might I prevent this in the future?
Let me know if this is enough information or not. Or if I need to clarify anything.
Thanks