Issues with installation in IPv6 environment

Elastic Stack Elasticsearch
1

Hi,

I am installing elasticsearch in a kubernetes environment with IPv6 network. Using helm charts from https://github.com/elastic/helm-charts/tree/master/elasticsearch.

  1. I configured network.host as ["::"], but with this setting, the masters do not form cluster. All 3 masters fail with masterNotDiscoveredException.
> {"level":"INFO","logger":"o.e.t.TransportService",**"log":"publish_address {127.0.0.1:9300}, bound_addresses {[::]:9300}"}**
> {"level":"INFO","logger":"o.e.b.BootstrapChecks","log":"bound or publishing to a non-loopback address, enforcing bootstrap checks"}
> {"level":"WARN","logger":"o.e.c.c.ClusterFormationFailureHelper","log":"master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elk-es-master-0, elk-es-master-1, elk-es-master-2] to bootstrap a cluster: have discovered []; discovery will continue using [[2019:280:4000:2001:9473:52ea:17a3:7b60]:9300, [2019:280:4000:2001:9496:22ed:bb1c:dd50]:9300, [2019:280:4000:2001:c262:ac59:17ae:fa66]:9300] from hosts providers and [{elk-es-master-0}{VbeA4QrPRv2YsNF1GA0gAQ}{0fpINSVoQSqhRc4N3OPJCg}{127.0.0.1}{127.0.0.1:9300}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"}
> {"level":"WARN","logger":"o.e.n.Node","log":"timed out while waiting for initial discovery state - timeout: 30s"}
> {"level":"INFO","logger":"o.e.h.AbstractHttpServerTransport","log":"publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}"}
> {"level":"INFO","logger":"o.e.n.Node","log":"started"}
> {"level":"WARN","logger":"r.suppressed","log":"path: /_cluster/health, params: {wait_for_status=green, timeout=1s}"}
> org.elasticsearch.discovery.MasterNotDiscoveredException: null
>         at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.0.1.jar:7.0.1]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
>         at java.lang.Thread.run(Thread.java:834) [?:?]

This is the network interfaces info from inside the container:

> [elasticsearch@elk-es-master-0 ~]$ ifconfig
> **eth0**: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet6 2019:280:4000:2001:9473:52ea:17a3:7b60  prefixlen 128  scopeid 0x0<global>
>         inet6 fe80::50c0:4fff:fe95:e035  prefixlen 64  scopeid 0x20<link>
>         ether 52:c0:4f:95:e0:35  txqueuelen 0  (Ethernet)
>         RX packets 33854  bytes 3342419 (3.1 MiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 33847  bytes 3341284 (3.1 MiB)
>         TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0
> 
> lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
>         inet 127.0.0.1  netmask 255.0.0.0
>         inet6 ::1  prefixlen 128  scopeid 0x10<host>
>         loop  txqueuelen 1000  (Local Loopback)
>         RX packets 271244  bytes 13308286 (12.6 MiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 271244  bytes 13308286 (12.6 MiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
  1. So, next I configured network.host as "_ eth0_:ipv6". With this all 3 master pods fail with the error
> "o.e.b.ElasticsearchUncaughtExceptionHandler","log":"uncaught exception in thread [main]"}
> org.elasticsearch.bootstrap.StartupException: BindTransportException[**Failed to resolve host [_eth0_:ipv6]]; nested: UnknownHostException[_eth0_:ipv6: Name or service not** known];
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.1.jar:7.0.1]
>         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.1.jar:7.0.1]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.1.jar:7.0.1]
> Caused by: org.elasticsearch.transport.BindTransportException: Failed to resolve host [_eth0_:ipv6]
>         at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:361) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:136) ~[?:?]
>         at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:61) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.transport.TransportService.doStart(TransportService.java:228) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:61) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.node.Node.start(Node.java:662) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:263) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:334) ~[elasticsearch-7.0.1.jar:7.0.1]
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.1.jar:7.0.1]
>         ... 6 more

Please let me know what would be right configuration for network_host & are any other network settings needed for IPv6 environment?

Thanks,
Shivani

2

Please could you share your whole elasticsearch.yml file?

Also, please format your posts properly, using the </> button for preformatted text, and checking that the preview looks readable. I will fix your post above.

3

Thanks @DavidTurner. My elasticsearch.yaml looks like this

cluster:
  name: elk-es
  initial_master_nodes: elk-es-master-0,elk-es-master-1,elk-es-master-2

node:
  master: true
  data: true
  name: elk-es-master-0
  ingest: true
  max_local_storage_nodes: 1

network.host: [::]

path:
  data: /data/data
  logs: /data/log
  repo: /data/elasticsearch

#bootstrap.memory_lock: true
http:
  compression: true
  cors:
    enabled: true
    allow-origin: *

discovery:
  seed_hosts: elasticsearch-master-headless
4

Thanks. I think

network.host: "_eth0:ipv6_"

should work for you, assuming that this picks the addresses to which elasticsearch-master-headless resolves. Could you try that (NB not _eth0_:ipv6)?

5

Thanks @DavidTurner :+1: Both of the below configurations worked for me -
network.host: "_eth0:ipv6_"
network.host: "_global:ipv6_"

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.