ECK- Discovery GCE plugin not working in GKE cluster

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi All,

I am trying to install gce discovery plugin in elasticsearch but unable to set the following parameters

# get the IP address from network interface 1
network.host: _gce:privateIp:1_
# Using GCE internal hostname
network.host: _gce:hostname_
# shortcut for _gce:privateIp:0_ (recommended)
network.host: _gce_

I am getting following error when i am giving any of the above config in elastic search.yml

ig", "node.name": "elasticsearch-config-es-master-0", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: BindTransportException[Failed to resolve host [_gce_]]; nested: IOException[IOException caught when fetching I
netAddress from [network-interfaces/0/ip]]; nested: IOException[failed to fetch metadata from [http://metadata.google.internal/computeMetadata/v1/instance/network-interface
s/0/ip]]; nested: HttpResponseException[404 Not Found",
"Not Found",
"];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.0.jar:7.4.0]",
"Caused by: org.elasticsearch.transport.BindTransportException: Failed to resolve host [_gce_]",
"at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:341) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:139) ~[?:?]",
"at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.doStart(SecurityNetty4Transport.java:81) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport.doStart(SecurityNetty4ServerTransport.java:43) ~[?:?]",
"at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.TransportService.doStart(TransportService.java:230) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.node.Node.start(Node.java:695) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]",
"... 6 more",

My elasticsearch.yaml seems to be picking up discoverygce setting for my projects and zone, but unable to pickup any of the network.host properties

cloud:
  gce:
    project_id: <PROJECT-ID>
    zone:
    - us-east4-a
    - us-east4-b
    - us-east4-c
cluster:
  name: elasticsearch-config
discovery:
  seed_providers: gce
http:
  compression: true
  compression_level: 9
network:
  host: 0.0.0.0
  publish_host: ${POD_IP}

Am i missing something, this cluster is on GKE.

2

Hey @aman26ps, is there a particular reason why you want to setup the gce discovery plugin? Everything should just work out of the box on GKE, you shouldn't have to setup any discovery plugin.

3

Hey sebgl, i want to use the host network for my es pods, by default my pod will be spunned up using pod network, i would like to use host interface of whichever node it chooses to spun up on.

4

Can you share your entire elasticsearch yaml spec, and the entire resulting elasticsearch.yaml configuration file?

5

Sure @sebgl

Please find my elastic-cr spec:

 #spec:
      #minAvailable: 2
      #selector:
        #matchLabels:
          #elasticsearch.k8s.elastic.co/cluster-name: elasticsearch-config
      config:
        # most Elasticsearch configuration parameters are possible to set, e.g: node.attr.attr_name: attr_value
        node.master: true
        node.data: false
        node.ingest: false
        node.ml: true
        # Uncomment this parameter you can specify which topology you will like to choose , hot-warm
        #node.attr.data: hot
        # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
        node.store.allow_mmap: false
        # Uncomment these two parameters if you wish to specify which zone es node should run in
        #node.attr.zone: europe-west3-a
        #cluster.routing.allocation.awareness.attributes: zone
        # for release 7.4.0 and 7.6.0 uncomment following line for filerealm config
        xpack.security.authc.realms.file.file1.order: 0
        # for release 6.8.8 uncomment following line for filerealm config
        #xpack.security.authc.realms.file1.type: file
        #xpack.security.authc.realms.file1.order: 0
        http.compression: true
        http.compression_level: 9
        cloud:
          gce:
              project_id: <project-id>
              zone: ["us-east4-a", "us-east4-b", "us-east4-c"]
        discovery:
              seed_providers: gce
        # get the IP address from network interface 1
        #network.host: _gce:privateIp:1_
        # # Using GCE internal hostname
        network.host: _gce:hostname_
        # # shortcut for _gce:privateIp:0_ (recommended)
        #network.host: _gce_

the elasticsearch.yml inside pod:

cloud:
  gce:
    project_id: <project-id>
    zone:
    - us-east4-a
    - us-east4-b
    - us-east4-c
cluster:
  name: elasticsearch-config
discovery:
  seed_providers: gce
http:
  compression: true
  compression_level: 9
network:
  host: 0.0.0.0
  publish_host: ${POD_IP}
node:
  data: true
  ingest: true
  master: false
  name: ${POD_NAME}
  store:
    allow_mmap: false
path:
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs
xpack:
  security:
    authc:
      realms:
        file:
          file1:
            order: 0
        native:
          native1:
            order: -99
      reserved_realm:
        enabled: "false"
    enabled: "true"
6

Use case to use discovery-gce plugin on gke = We are looking for running some of ES data nodes on host network (Pod needs to have host IP )so that we can join them to a running ES cluster to migrate ES VM to VM on GKE.

7

Looks like another issue has been created for this: ECK - can we run ECK operator in privileged mode.

8

yes @sebgl for time being we have decided to drop this requirement because as we do more research on this topic we find that the discovery-gce is not really meant for GKE where networking is managed by google, thanks for your help