Thank you.
I am not familiar with network interfaces. I will communicate the team responsible for server/VM configurations.
If you have any documentation/reference on setting network interface on a linux server it would be appreciated.
Here are the result for ifconfig on the server
> cni-podman0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet 10.88.0.1 netmask 255.255.0.0 broadcast 10.88.255.255
> inet6 fe80::9845:2cff:fe7f:8915 prefixlen 64 scopeid 0x20<link>
> ether 9a:45:2c:7f:89:15 txqueuelen 1000 (Ethernet)
> RX packets 104160 bytes 96982793 (92.4 MiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 105971 bytes 48620356 (46.3 MiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet 172.16.82.115 netmask 255.255.254.0 broadcast 172.16.83.255
> ether 00:50:56:be:42:e0 txqueuelen 1000 (Ethernet)
> RX packets 1034280 bytes 394301449 (376.0 MiB)
> RX errors 0 dropped 314 overruns 0 frame 0
> TX packets 628514 bytes 222824824 (212.5 MiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
> inet 127.0.0.1 netmask 255.0.0.0
> inet6 ::1 prefixlen 128 scopeid 0x10<host>
> loop txqueuelen 1000 (Local Loopback)
> RX packets 30 bytes 1980 (1.9 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 30 bytes 1980 (1.9 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> veth45768a64: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet6 fe80::d8e5:10ff:feeb:ffdf prefixlen 64 scopeid 0x20<link>
> ether da:e5:10:eb:ff:df txqueuelen 0 (Ethernet)
> RX packets 273 bytes 29901 (29.2 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 7524 bytes 472941 (461.8 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> veth5617377d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet6 fe80::c4ce:c6ff:feca:e1b5 prefixlen 64 scopeid 0x20<link>
> ether c6:ce:c6:ca:e1:b5 txqueuelen 0 (Ethernet)
> RX packets 232 bytes 28378 (27.7 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 270 bytes 48220 (47.0 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
As you see, inet 172.16.82.115 is defined in ens192
Should I use broadcast 172.16.83.255?
or the Podman one (inet 10.88.0.1)
By "on the server" do you mean you ran ifconfig within the container that holds Elasticsearch? Elasticsearch is saying that it doesn't have access to any interfaces with address 172.16.82.115.
I cannot run ifconfig in the container image. I do not know if it is a question of installation or permission and since the container keeps restarting as soon as it detects an error, I do not have time to investigate.
I run podman inspect es01 and I get this:
> {
> "Id": "3fbd2f05b8cb7b4f8205db3dbe90bbfe6c15d63c3edeadd9970193b27a68b831",
> "Created": "2022-03-02T12:48:44.421171848-05:00",
> "Path": "/bin/tini",
> "Args": [
> "--",
> "/usr/local/bin/docker-entrypoint.sh",
> "eswrapper"
> ],
> "State": {
> "OciVersion": "1.0.2-dev",
> "Status": "running",
> "Running": true,
> "Paused": false,
> "Restarting": false,
> "OOMKilled": false,
> "Dead": false,
> "Pid": 2152645,
> "ConmonPid": 2152633,
> "ExitCode": 0,
> "Error": "",
> "StartedAt": "2022-03-02T12:48:44.926132745-05:00",
> "FinishedAt": "0001-01-01T00:00:00Z",
> "Healthcheck": {
> "Status": "",
> "FailingStreak": 0,
> "Log": null
> }
> },
> "Image": "fef75c42834fe3e0ddec4be2a6f929bbe5b1d7a8d41230c05e1485545f8d9433",
> "ImageName": "docker.elastic.co/elasticsearch/elasticsearch:8.0.0",
> "Rootfs": "",
> "Pod": "7d884d53793cb1c35f6d93b4f64b5caa7c818c9910d078c94f8df9ee92d3f0b4",
> "ResolvConfPath": "/run/containers/storage/overlay-containers/2724e34f8e580c5d202c63d38ade03f6acd5c06fa74758107d706072d6b462ba/userdata/resolv.conf",
> "HostnamePath": "/run/containers/storage/overlay-containers/3fbd2f05b8cb7b4f8205db3dbe90bbfe6c15d63c3edeadd9970193b27a68b831/userdata/hostname",
> "HostsPath": "/run/containers/storage/overlay-containers/2724e34f8e580c5d202c63d38ade03f6acd5c06fa74758107d706072d6b462ba/userdata/hosts",
> "StaticDir": "/var/lib/containers/storage/overlay-containers/3fbd2f05b8cb7b4f8205db3dbe90bbfe6c15d63c3edeadd9970193b27a68b831/userdata",
> "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/3fbd2f05b8cb7b4f8205db3dbe90bbfe6c15d63c3edeadd9970193b27a68b831/userdata/config.json",
> "OCIRuntime": "runc",
> "ConmonPidFile": "/run/containers/storage/overlay-containers/3fbd2f05b8cb7b4f8205db3dbe90bbfe6c15d63c3edeadd9970193b27a68b831/userdata/conmon.pid",
> "PidFile": "/run/containers/storage/overlay-containers/3fbd2f05b8cb7b4f8205db3dbe90bbfe6c15d63c3edeadd9970193b27a68b831/userdata/pidfile",
> "Name": "es01",
> "RestartCount": 0,
> "Driver": "overlay",
> "MountLabel": "system_u:object_r:container_file_t:s0:c164,c969",
> "ProcessLabel": "system_u:system_r:container_t:s0:c164,c969",
> "AppArmorProfile": "",
> "EffectiveCaps": null,
> "BoundingCaps": [
> "CAP_CHOWN",
> "CAP_DAC_OVERRIDE",
> "CAP_FOWNER",
> "CAP_FSETID",
> "CAP_KILL",
> "CAP_NET_BIND_SERVICE",
> "CAP_NET_RAW",
> "CAP_SETFCAP",
> "CAP_SETGID",
> "CAP_SETPCAP",
> "CAP_SETUID",
> "CAP_SYS_CHROOT"
> ],
> "ExecIDs": [],
> "GraphDriver": {
> "Name": "overlay",
> "Data": {
> "LowerDir": "/var/lib/containers/storage/overlay/3d4abcd836122cf7f4db45068ae907ab3c5d61a22b0a72b8d287fcef16fc4bb7/diff:/var/lib/containers/storage/overlay/c964d8d65b6d97ccb40031e434894a49a94bd15cff02f28f3b107db58c803c3c/diff:/var/lib/containers/storage/overlay/8865f716956a8e871c4926598cbd3c6e1df74b7059c2d33d51350f1e57cf67ce/diff:/var/lib/containers/storage/overlay/8f7d00ebc5692e4b32cc4dd92132f90a3711d344fdbc81b2efa01997a0f7c5d6/diff:/var/lib/containers/storage/overlay/531ada9facdda2aee8add0d66b2db91b99c668051672feef6297cf24ea228430/diff:/var/lib/containers/storage/overlay/2c958ea9a23257bf7aeb8f61e848f959f79606667df6a2a766e2eacef262a122/diff:/var/lib/containers/storage/overlay/86e21de91c76321e530ada9f5bb85bc2634d7da864d8589b79d5ff115dc5b3ab/diff:/var/lib/containers/storage/overlay/16ae6025f2b3344fe5b786115c517685fa7ad6b3c291a4097a9b6f8a5fd33899/diff:/var/lib/containers/storage/overlay/04bc6da929fadd9bfbe575d99eb0dfef06f588e4cbe17f4fddb451ec7434a919/diff:/var/lib/containers/storage/overlay/36ffdceb4c77bf34325fb695e64ea447f688797f2f1e3af224c29593310578d2/diff",
> "MergedDir": "/var/lib/containers/storage/overlay/af52f33c506d8b8f8736507a753db1f2a320ecfae1a8c6f4b795a408700f8954/merged",
> "UpperDir": "/var/lib/containers/storage/overlay/af52f33c506d8b8f8736507a753db1f2a320ecfae1a8c6f4b795a408700f8954/diff",
> "WorkDir": "/var/lib/containers/storage/overlay/af52f33c506d8b8f8736507a753db1f2a320ecfae1a8c6f4b795a408700f8954/work"
> }
> },
> "Mounts": [
> {
> "Type": "bind",
> "Source": "/var/elasticsearch/elasticsearch-data/es01",
> "Destination": "/usr/share/elasticsearch/data",
> "Driver": "",
> "Mode": "",
> "Options": [
> "rbind"
> ],
> "RW": true,
> "Propagation": "rprivate"
> },
> {
> "Type": "bind",
> "Source": "/var/elasticsearch/elasticsearch-config/es01",
> "Destination": "/usr/share/elasticsearch/config",
> "Driver": "",
> "Mode": "",
> "Options": [
> "rbind"
> ],
> "RW": true,
> "Propagation": "rprivate"
> },
> {
> "Type": "bind",
> "Source": "/var/elasticsearch/elasticsearch-certs",
> "Destination": "/usr/share/elasticsearch/config/certs",
> "Driver": "",
> "Mode": "",
> "Options": [
> "rbind"
> ],
> "RW": true,
> "Propagation": "rprivate"
> }
> ],
> "Dependencies": [
> "2724e34f8e580c5d202c63d38ade03f6acd5c06fa74758107d706072d6b462ba"
> ],
> "NetworkSettings": {
> "EndpointID": "",
> "Gateway": "10.88.0.1",
> "IPAddress": "10.88.0.22",
> "IPPrefixLen": 16,
> "IPv6Gateway": "",
> "GlobalIPv6Address": "",
> "GlobalIPv6PrefixLen": 0,
> "MacAddress": "2a:c5:5b:c1:80:b1",
> "Bridge": "",
> "SandboxID": "",
> "HairpinMode": false,
> "LinkLocalIPv6Address": "",
> "LinkLocalIPv6PrefixLen": 0,
> "Ports": {
> "5601/tcp": [
> {
> "HostIp": "",
> "HostPort": "8080"
> }
> ],
> "9200/tcp": [
> {
> "HostIp": "",
> "HostPort": "9200"
> }
> ],
> "9300/tcp": [
> {
> "HostIp": "",
> "HostPort": "9300"
> }
> ],
> "9301/tcp": [
> {
> "HostIp": "",
> "HostPort": "9301"
> }
> ],
> "9302/tcp": [
> {
> "HostIp": "",
> "HostPort": "9302"
> }
> ],
> "9303/tcp": [
> {
> "HostIp": "",
> "HostPort": "9303"
> }
> ],
> "9304/tcp": [
> {
> "HostIp": "",
> "HostPort": "9304"
> }
> ],
> "9305/tcp": [
> {
> "HostIp": "",
> "HostPort": "9305"
> }
> ]
> },
> "SandboxKey": "/run/netns/cni-e6f4c6a7-13e0-2e4a-7043-3756934d067e",
> "Networks": {
> "podman": {
> "EndpointID": "",
> "Gateway": "10.88.0.1",
> "IPAddress": "10.88.0.22",
> "IPPrefixLen": 16,
> "IPv6Gateway": "",
> "GlobalIPv6Address": "",
> "GlobalIPv6PrefixLen": 0,
> "MacAddress": "2a:c5:5b:c1:80:b1",
> "NetworkID": "podman",
> "DriverOpts": null,
> "IPAMConfig": null,
> "Links": null
> }
> }
> },
> "ExitCommand": [
> "/usr/bin/podman",
> "--root",
> "/var/lib/containers/storage",
> "--runroot",
> "/run/containers/storage",
> "--log-level",
> "warning",
> "--cgroup-manager",
> "systemd",
> "--tmpdir",
> "/run/libpod",
> "--runtime",
> "runc",
> "--storage-driver",
> "overlay",
> "--storage-opt",
> "overlay.mountopt=nodev,metacopy=on",
> "--events-backend",
> "file",
> "container",
> "cleanup",
> "--rm",
> "3fbd2f05b8cb7b4f8205db3dbe90bbfe6c15d63c3edeadd9970193b27a68b831"
> ],
> "Namespace": "",
> "IsInfra": false,
> "Config": {
> "Hostname": "elasticsearch",
> "Domainname": "",
> "User": "elasticsearch:root",
> "AttachStdin": false,
> "AttachStdout": false,
> "AttachStderr": false,
> "Tty": false,
> "OpenStdin": false,
> "StdinOnce": false,
> "Env": [
> "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
> "TERM=xterm",
> "container=podman",
> "ELASTIC_CONTAINER=true",
> "ES_ZLIB_PATH=/usr/local/cloudflare-zlib/lib",
> "ES_JAVA_OPTS=-Xms1g -Xmx1g",
> "HOME=/usr/share/elasticsearch",
> "HOSTNAME=elasticsearch"
> ],
> "Cmd": [
> "eswrapper"
> ],
> "Image": "docker.elastic.co/elasticsearch/elasticsearch:8.0.0",
> "Volumes": null,
> "WorkingDir": "/usr/share/elasticsearch",
> "Entrypoint": "/bin/tini -- /usr/local/bin/docker-entrypoint.sh",
> "OnBuild": null,
> "Labels": {
> "PODMAN_SYSTEMD_UNIT": "container-es01.service",
> "org.label-schema.build-date": "2022-02-03T22:52:11.916594404Z",
> "org.label-schema.license": "Elastic-License-2.0",
> "org.label-schema.name": "Elasticsearch",
> "org.label-schema.schema-version": "1.0",
> "org.label-schema.url": "https://www.elastic.co/products/elasticsearch",
> "org.label-schema.usage": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
> "org.label-schema.vcs-ref": "1b6a7ece17463df5ff54a3e1302d825889aa1161",
> "org.label-schema.vcs-url": "https://github.com/elastic/elasticsearch",
> "org.label-schema.vendor": "Elastic",
> "org.label-schema.version": "8.0.0",
> "org.opencontainers.image.created": "2022-02-03T22:52:11.916594404Z",
> "org.opencontainers.image.documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
> "org.opencontainers.image.licenses": "Elastic-License-2.0",
> "org.opencontainers.image.revision": "1b6a7ece17463df5ff54a3e1302d825889aa1161",
> "org.opencontainers.image.source": "https://github.com/elastic/elasticsearch",
> "org.opencontainers.image.title": "Elasticsearch",
> "org.opencontainers.image.url": "https://www.elastic.co/products/elasticsearch",
> "org.opencontainers.image.vendor": "Elastic",
> "org.opencontainers.image.version": "8.0.0"
> },
> "Annotations": {
> "io.container.manager": "libpod",
> "io.kubernetes.cri-o.ContainerType": "container",
> "io.kubernetes.cri-o.Created": "2022-03-02T12:48:44.421171848-05:00",
> "io.kubernetes.cri-o.SandboxID": "7d884d53793cb1c35f6d93b4f64b5caa7c818c9910d078c94f8df9ee92d3f0b4",
> "io.kubernetes.cri-o.TTY": "false",
> "io.podman.annotations.autoremove": "TRUE",
> "io.podman.annotations.cid-file": "/run/container-es01.service.ctr-id",
> "io.podman.annotations.init": "FALSE",
> "io.podman.annotations.privileged": "FALSE",
> "io.podman.annotations.publish-all": "FALSE",
> "org.opencontainers.image.stopSignal": "15"
> },
> "StopSignal": 15,
> "CreateCommand": [
> "/usr/bin/podman",
> "run",
> "--cidfile=/run/container-es01.service.ctr-id",
> "--sdnotify=conmon",
> "--cgroups=no-conmon",
> "--group-add",
> "0",
> "--rm",
> "--pod-id-file",
> "/run/pod-elasticsearch.pod-id",
> "-d",
> "--replace",
> "--name=es01",
> "-e",
> "ES_JAVA_OPTS=-Xms1g -Xmx1g",
> "-v",
> "/var/elasticsearch/elasticsearch-data/es01:/usr/share/elasticsearch/data:z",
> "-v",
> "/var/elasticsearch/elasticsearch-config/es01:/usr/share/elasticsearch/config:z",
> "-v",
> "/var/elasticsearch/elasticsearch-certs:/usr/share/elasticsearch/config/certs:z",
> "docker.elastic.co/elasticsearch/elasticsearch:8.0.0"
> ],
> "Umask": "0022",
> "Timeout": 0,
> "StopTimeout": 10
> },
> "HostConfig": {
> "Binds": [
> "/var/elasticsearch/elasticsearch-data/es01:/usr/share/elasticsearch/data:rw,rprivate,rbind",
> "/var/elasticsearch/elasticsearch-config/es01:/usr/share/elasticsearch/config:rw,rprivate,rbind",
> "/var/elasticsearch/elasticsearch-certs:/usr/share/elasticsearch/config/certs:rw,rprivate,rbind"
> ],
> "CgroupManager": "systemd",
> "CgroupMode": "host",
> "ContainerIDFile": "/run/container-es01.service.ctr-id",
> "LogConfig": {
> "Type": "k8s-file",
> "Config": null,
> "Path": "/var/lib/containers/storage/overlay-containers/3fbd2f05b8cb7b4f8205db3dbe90bbfe6c15d63c3edeadd9970193b27a68b831/userdata/ctr.log",
> "Tag": "",
> "Size": "0B"
> },
> "NetworkMode": "container:2724e34f8e580c5d202c63d38ade03f6acd5c06fa74758107d706072d6b462ba",
> "PortBindings": {},
> "RestartPolicy": {
> "Name": "",
> "MaximumRetryCount": 0
> },
> "AutoRemove": true,
> "VolumeDriver": "",
> "VolumesFrom": null,
> "CapAdd": [],
> "CapDrop": [
> "CAP_AUDIT_WRITE",
> "CAP_MKNOD"
> ],
> "Dns": [],
> "DnsOptions": [],
> "DnsSearch": [],
> "ExtraHosts": [],
> "GroupAdd": [
> "0"
> ],
> "IpcMode": "container:2724e34f8e580c5d202c63d38ade03f6acd5c06fa74758107d706072d6b462ba",
> "Cgroup": "",
> "Cgroups": "default",
> "Links": null,
> "OomScoreAdj": 0,
> "PidMode": "private",
> "Privileged": false,
> "PublishAllPorts": false,
> "ReadonlyRootfs": false,
> "SecurityOpt": [],
> "Tmpfs": {},
> "UTSMode": "container:2724e34f8e580c5d202c63d38ade03f6acd5c06fa74758107d706072d6b462ba",
> "UsernsMode": "",
> "ShmSize": 65536000,
> "Runtime": "oci",
> "ConsoleSize": [
> 0,
> 0
> ],
> "Isolation": "",
> "CpuShares": 0,
> "Memory": 0,
> "NanoCpus": 0,
> "CgroupParent": "machine.slice/machine-libpod_pod_7d884d53793cb1c35f6d93b4f64b5caa7c818c9910d078c94f8df9ee92d3f0b4.slice",
> "BlkioWeight": 0,
> "BlkioWeightDevice": null,
> "BlkioDeviceReadBps": null,
> "BlkioDeviceWriteBps": null,
> "BlkioDeviceReadIOps": null,
> "BlkioDeviceWriteIOps": null,
> "CpuPeriod": 0,
> "CpuQuota": 0,
> "CpuRealtimePeriod": 0,
> "CpuRealtimeRuntime": 0,
> "CpusetCpus": "",
> "CpusetMems": "",
> "Devices": [],
> "DiskQuota": 0,
> "KernelMemory": 0,
> "MemoryReservation": 0,
> "MemorySwap": 0,
> "MemorySwappiness": 0,
> "OomKillDisable": false,
> "PidsLimit": 2048,
> "Ulimits": [
> {
> "Name": "RLIMIT_NOFILE",
> "Soft": 1048576,
> "Hard": 1048576
> },
> {
> "Name": "RLIMIT_NPROC",
> "Soft": 4194304,
> "Hard": 4194304
> }
> ],
> "CpuCount": 0,
> "CpuPercent": 0,
> "IOMaximumIOps": 0,
> "IOMaximumBandwidth": 0,
> "CgroupConf": null
> }
> }
It seems the network address of the container is:
>"NetworkSettings": {
> "EndpointID": "",
> "Gateway": "10.88.0.1",
> "IPAddress": "10.88.0.22",
Should I use 10.88.0.22?
Sounds like it, if that's the address of the network interface within the container.
Sorry that I can't be more helpful here, these questions are all very much about your infrastructure & networking setup rather than anything to do with Elasticsearch. Container networking is tricky, there's very many subtleties, and it might be better to ask a local expert for help sorting these things out since they will understand better. The setup Elasticsearch needs is described in the docs I shared earlier, and if it doesn't work for a few minutes then it should start logging some increasingly helpful hints about the problem.
I should point out that I don't think anything has changed in this area between 7.17.1 and 8.0.0. I can imagine ways that it might work out of sheer luck sometimes.
I understand.
I am looking with IT to better clear this out.
There is not much information available on running Elasticsearch in RHEL podman containers in production mode. Would be nice to have examples of setup including multiple containers distributed on multiple servers/VM.
Will share findings on my side.
I have succeeded to mount Elasticsearch cluster in podman by
1- setting a unique transport.port to each node,
2- set a static ip address in Elasticsearch.yml ( network.host: 10.88.0.10)
3- define this ip address at the pod level (podman pod create with --ip 10.88.0.10)
It works well and I do not have anymore the problem with master election.
Now I setup a second server with the same settings and try to make another node join the original cluster.
The pod in the second server runs on a diff ip (10.89.0.10) and it can run locally (when I set single discovery-mode)
There is a route between the 2 servers.
However if fails to join the cluster on the other server.
The problem is not that it cannot see the server nor the master node. It simply fails to join it.
The actual master in the cluster is 10.88.0.10:9301
Do you know what could be the problem?
> {"type": "server", "timestamp": "2022-03-07T13:36:00,381Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-docker-cluster", "node.name": "es04", "message": "master not discovered yet: have discovered [{es04}{f1ICFD7cTt2r4uEZEPf4aQ}{iVyKhh0BR4WXhhn5EFMvxw}{10.89.0.10}{10.89.0.10:9314}{d}, {es01}{CDPAYocARvC_f9FtrBwusg}{MkFqMV1yQ3-5Q6rUC-M9VQ}{10.88.0.10}{10.88.0.10:9301}{dimt}, {es02}{hrXk0AOHRWqTiLmzZdbghw}{96puQmx1Te-MJ3lhCp3Sfg}{10.88.0.10}{10.88.0.10:9302}{dimt}, {es03}{SofMGtv0TmCPQdFltXAMOQ}{dGW2nqDzSR2kmfaFugG_Vg}{10.88.0.10}{10.88.0.10:9303}{dimt}]; discovery will continue using [10.88.0.10:9301, 10.88.0.10:9302, 10.88.0.10:9303] from hosts providers and [] from last-known cluster state; node term 308, last-accepted version 0 in term 0" }
> {"type": "server", "timestamp": "2022-03-07T13:36:05,061Z", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "es-docker-cluster", "node.name": "es04", "message": "failed to join {es01}{CDPAYocARvC_f9FtrBwusg}{MkFqMV1yQ3-5Q6rUC-M9VQ}{10.88.0.10}{10.88.0.10:9301}{dimt}{xpack.installed=true} with JoinRequest{sourceNode={es04}{f1ICFD7cTt2r4uEZEPf4aQ}{iVyKhh0BR4WXhhn5EFMvxw}{10.89.0.10}{10.89.0.10:9314}{d}{xpack.installed=true}, minimumTerm=308, optionalJoin=Optional.empty}",
> "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [es01][10.88.0.10:9301][internal:cluster/coordination/join]",
> "Caused by: org.elasticsearch.transport.ConnectTransportException: [es04][10.89.0.10:9314] connect_timeout[30s]",
> "at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1038) ~[elasticsearch-8.0.0.jar:8.0.0]",
> "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717) ~[elasticsearch-8.0.0.jar:8.0.0]",
> "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]",
> "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]",
> "at java.lang.Thread.run(Thread.java:833) [?:?]"] }
> {"type": "server", "timestamp": "2022-03-07T13:36:10,382Z", "level": "WARN", "component": "o.e.c.c.JoinHelper", "cluster.name": "es-docker-cluster", "node.name": "es04", "message": "last failed join attempt was 5.3s ago, failed to join {es01}{CDPAYocARvC_f9FtrBwusg}{MkFqMV1yQ3-5Q6rUC-M9VQ}{10.88.0.10}{10.88.0.10:9301}{dimt}{xpack.installed=true} with JoinRequest{sourceNode={es04}{f1ICFD7cTt2r4uEZEPf4aQ}{iVyKhh0BR4WXhhn5EFMvxw}{10.89.0.10}{10.89.0.10:9314}{d}{xpack.installed=true}, minimumTerm=308, optionalJoin=Optional.empty}",
> "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [es01][10.88.0.10:9301][internal:cluster/coordination/join]",
> "Caused by: org.elasticsearch.transport.ConnectTransportException: [es04][10.89.0.10:9314] connect_timeout[30s]",
> "at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1038) ~[elasticsearch-8.0.0.jar:8.0.0]",
> "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717) ~[elasticsearch-8.0.0.jar:8.0.0]",
> "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
> "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
> "at java.lang.Thread.run(Thread.java:833) [?:?]"] }
> {"type": "server", "timestamp": "2022-03-07T13:36:10,383Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-docker-cluster", "node.name": "es04", "message": "master not discovered yet: have discovered [{es04}{f1ICFD7cTt2r4uEZEPf4aQ}{iVyKhh0BR4WXhhn5EFMvxw}{10.89.0.10}{10.89.0.10:9314}{d}, {es01}{CDPAYocARvC_f9FtrBwusg}{MkFqMV1yQ3-5Q6rUC-M9VQ}{10.88.0.10}{10.88.0.10:9301}{dimt}, {es02}{hrXk0AOHRWqTiLmzZdbghw}{96puQmx1Te-MJ3lhCp3Sfg}{10.88.0.10}{10.88.0.10:9302}{dimt}, {es03}{SofMGtv0TmCPQdFltXAMOQ}{dGW2nqDzSR2kmfaFugG_Vg}{10.88.0.10}{10.88.0.10:9303}{dimt}]; discovery will continue using [10.88.0.10:9301, 10.88.0.10:9302, 10.88.0.10:9303] from hosts providers and [] from last-known cluster state; node term 308, last-accepted version 0 in term 0" }
> {"type": "server", "timestamp": "2022-03-07T13:36:20,383Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-docker-cluster", "node.name": "es04", "message": "master not discovered yet: have discovered [{es04}{f1ICFD7cTt2r4uEZEPf4aQ}{iVyKhh0BR4WXhhn5EFMvxw}{10.89.0.10}{10.89.0.10:9314}{d}, {es01}{CDPAYocARvC_f9FtrBwusg}{MkFqMV1yQ3-5Q6rUC-M9VQ}{10.88.0.10}{10.88.0.10:9301}{dimt}, {es02}{hrXk0AOHRWqTiLmzZdbghw}{96puQmx1Te-MJ3lhCp3Sfg}{10.88.0.10}{10.88.0.10:9302}{dimt}, {es03}{SofMGtv0TmCPQdFltXAMOQ}{dGW2nqDzSR2kmfaFugG_Vg}{10.88.0.10}{10.88.0.10:9303}{dimt}]; discovery will continue using [10.88.0.10:9301, 10.88.0.10:9302, 10.88.0.10:9303] from hosts providers and [] from last-known cluster state; node term 308, last-accepted version 0 in term 0" }
This is still a connectivity problem. From the docs linked above:
The only requirements are that each node must be... accessible at its transport publish address by all other nodes in its cluster.
Your setup does not currently satisfy this.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.