# Elastic application is down because of unknown node.remote\_cluster\_client **URL:** \ **Category:** Elasticsearch\ **Created:** [May 9, 2021, 6:51am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484 "2021-05-09T06:51:48Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Dawood](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Dawood](https://discuss.elastic.co/u/Dawood)\ **Post date:** [May 9, 2021, 6:51am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/1 "2021-05-09T06:51:48Z")
![elasticsearch](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a98e878e44961f42712967c3b2cd0d63ffd3432b.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 9, 2021, 9:19am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/2 "2021-05-09T09:19:21Z")
Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable. Instead, paste the text and format it with `` icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer. It would be great if you could update your post to solve this. Here you probably need to fix/remove the setting you added. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 10, 2021, 12:21am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/3 "2021-05-10T00:21:39Z")
It's also appreciated if you don't just post a topic with a single - hard to read as David mentions - image, with nothing else. Are you just sharing your log or are you asking for assistance? ---
**Author:** ![Dawood](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Dawood](https://discuss.elastic.co/u/Dawood)\ **Post date:** [May 12, 2021, 9:38am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/4 "2021-05-12T09:38:14Z")
I got to this situation after i got errors in elasticsearch logs that related to memory: ```` ```stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2] `Preformatted text``Preformatted text`[10.42.0.144:9300][internal:cluster/coordination/join]", "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower", "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:467) [elasticsearch-7.6.1.jar:7.6.1]", "at java.util.HashMap$Values.forEach(HashMap.java:981) [?:?]",``` ```` Then i make some research in the net and found that i should change "java options" in elasticsearch.yaml. So i changed them to: esJavaOpts: "-Xmx8g -Xms8g" instead of esJavaOpts: "-Xmx6g -Xms6g". After that change i got the error that i mentioned in the previous message: > ``` > {"type": "server", "timestamp": "2021-05-12T08:34:02,871Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "uncaught exception in thread [main]", > "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [node.remote_cluster_client] please check that any required plugins are installed, or check the breaking changes documentation for removed settings", > "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.1.jar:7.6.1]", > > ``` Note: Elasticsearch app is part of ELK application that runs on k8s( Rancher) based on 3 servers with the following resources: 8 CPU and 32 RAM and NFS storage. Let me know if other information such as yaml files for the 3 applications will help to find the root cause of that issue. Appreciate your help 🙂 ---
**Author:** ![Dawood](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Dawood](https://discuss.elastic.co/u/Dawood)\ **Post date:** [May 12, 2021, 9:40am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/5 "2021-05-12T09:40:03Z")
I am asking for assistance. Can you please look at my reply to David. Thanks 🙂 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 12, 2021, 10:11am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/6 "2021-05-12T10:11:20Z")
You probably need to remove this setting `node.remote_cluster_client`. ---
**Author:** ![Dawood](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Dawood](https://discuss.elastic.co/u/Dawood)\ **Post date:** [May 12, 2021, 10:52am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/7 "2021-05-12T10:52:38Z")
I did not added to the yaml file. Wired this setting was added automatically, after deleting it from k8s yaml everything works fine. I see something in the Rancher (k8s) that i Can not understand if it is correct or not: > [docker.elastic.co/elasticsearch/elasticsearch:7.6.1](http://docker.elastic.co/elasticsearch/elasticsearch:7.6.1) + 1 image I used the chart yamle from:[https://github.com/elastic/helm-charts/blob/master/logstash/values.yaml](https://github.com/elastic/helm-charts/blob/master/logstash/values.yaml) with some minor modifications. Deploy the app with new values and got 2 images in k8s. Any idea? Thanks for you help ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 12, 2021, 1:03pm UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/8 "2021-05-12T13:03:30Z")
> [@Dawood](#): > > Any idea? No. Not my area sorry. ☹ ---
**Author:** ![Dawood](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Dawood](https://discuss.elastic.co/u/Dawood)\ **Post date:** [May 13, 2021, 5:27am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/9 "2021-05-13T05:27:02Z")
> [@Dawood](#): > > I got to this situation after i got errors in elasticsearch logs that related to memory: Can you just please explain the error part of the memory (see above). And how could i resolve it. Cause i am pretty sure that it will happen again and broke the application ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 13, 2021, 6:22am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/10 "2021-05-13T06:22:39Z")
Could you share your elasticsearch.yml configuration? ---
**Author:** ![Dawood](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Dawood](https://discuss.elastic.co/u/Dawood)\ **Post date:** [May 16, 2021, 7:26am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/11 "2021-05-16T07:26:55Z")
```auto --- clusterName: "elasticsearch" nodeGroup: "master" # The service that non master groups will try to connect to when joining the cluster # This should be set to clusterName + "-" + nodeGroup for your master group masterService: "" # Elasticsearch roles that will be applied to this nodeGroup # These will be set as environment variables. E.g. node.master=true roles: master: "true" ingest: "true" data: "true" replicas: 3 minimumMasterNodes: 2 esMajorVersion: "" # Allows you to add any config files in /usr/share/elasticsearch/config/ # such as elasticsearch.yml and log4j2.properties esConfig: {} # elasticsearch.yml: | # key: # nestedkey: value # log4j2.properties: | # key = value # Extra environment variables to append to this nodeGroup # This will be appended to the current 'env:' key. You can use any of the kubernetes env # syntax here extraEnvs: [] # - name: MY_ENVIRONMENT_VAR # value: the_value_goes_here # Allows you to load environment variables from kubernetes secret or config map envFrom: [] # - secretRef: # name: env-secret # - configMapRef: # name: config-map # A list of secrets and their paths to mount inside the pod # This is useful for mounting certificates for security and for mounting # the X-Pack license secretMounts: [] # - name: elastic-certificates # secretName: elastic-certificates # path: /usr/share/elasticsearch/config/certs # defaultMode: 0755 image: "docker.elastic.co/elasticsearch/elasticsearch" imageTag: "7.6.1" imagePullPolicy: "IfNotPresent" podAnnotations: {} # iam.amazonaws.com/role: es-cluster # additionals labels labels: {} esJavaOpts: "-Xmx6g -Xms6g" resources: requests: cpu: "1000m" memory: "8Gi" limits: cpu: "1000m" memory: "10Gi" initResources: {} # limits: # cpu: "25m" # # memory: "128Mi" # requests: # cpu: "25m" # memory: "128Mi" sidecarResources: {} # limits: # cpu: "25m" # # memory: "128Mi" # requests: # cpu: "25m" # memory: "128Mi" networkHost: "0.0.0.0" volumeClaimTemplate: accessModes: ["ReadWriteOnce"] resources: requests: storage: 30Gi rbac: create: false serviceAccountAnnotations: {} serviceAccountName: "" podSecurityPolicy: create: false name: "" spec: privileged: true fsGroup: rule: RunAsAny runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - secret - configMap - persistentVolumeClaim persistence: enabled: true labels: # Add default labels for the volumeClaimTemplate fo the StatefulSet enabled: false annotations: {} extraVolumes: [] # - name: extras # emptyDir: {} extraVolumeMounts: [] # - name: extras # mountPath: /usr/share/extras # readOnly: true extraContainers: [] # - name: do-something # image: busybox # command: ['do', 'something'] extraInitContainers: [] # - name: do-something # image: busybox # command: ['do', 'something'] # This is the PriorityClass settings as defined in # https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass priorityClassName: "" # By default this will make sure two pods don't end up on the same node # Changing this to a region would allow you to spread pods across regions antiAffinityTopologyKey: "kubernetes.io/hostname" # Hard means that by default pods will only be scheduled if there are enough nodes for them # and that they will never end up on the same node. Setting this to soft will do this "best effort" antiAffinity: "hard" # This is the node affinity settings as defined in # https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature nodeAffinity: {} # The default is to deploy all pods serially. By setting this to parallel all pods are started at # the same time when bootstrapping the cluster podManagementPolicy: "Parallel" # The environment variables injected by service links are not used, but can lead to slow Elasticsearch boot times when # there are many services in the current namespace. # If you experience slow pod startups you probably want to set this to `false`. enableServiceLinks: true protocol: http httpPort: 9200 transportPort: 9300 service: labels: {} labelsHeadless: {} type: ClusterIP nodePort: "" annotations: {} httpPortName: http transportPortName: transport loadBalancerIP: "" loadBalancerSourceRanges: [] externalTrafficPolicy: "" updateStrategy: RollingUpdate # This is the max unavailable setting for the pod disruption budget # The default value of 1 will make sure that kubernetes won't allow more than 1 # of your pods to be unavailable during maintenance maxUnavailable: 1 podSecurityContext: fsGroup: 1000 runAsUser: 1000 securityContext: capabilities: drop: - ALL # readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 # How long to wait for elasticsearch to stop gracefully terminationGracePeriod: 120 sysctlVmMaxMapCount: 262144 readinessProbe: failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 3 timeoutSeconds: 5 # https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status clusterHealthCheckParams: "wait_for_status=green&timeout=1s" ## Use an alternate scheduler. ## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ ## schedulerName: "" imagePullSecrets: [] nodeSelector: {} tolerations: [] # Enabling this will publically expose your Elasticsearch instance. # Only enable this if you have security enabled on your cluster ingress: enabled: true annotations: {} # kubernetes.io/ingress.class: nginx # kubernetes.io/tls-acme: "true" path: / hosts: - qa-elk.labs.mlnx tls: [] # - secretName: chart-example-tls # hosts: # - chart-example.local nameOverride: "" fullnameOverride: "" # https://github.com/elastic/helm-charts/issues/63 masterTerminationFix: false lifecycle: {} # preStop: # exec: # command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"] # postStart: # exec: # command: # - bash # - -c # - | # #!/bin/bash # # Add a template to adjust number of shards/replicas # TEMPLATE_NAME=my_template # INDEX_PATTERN="logstash-*" # SHARD_COUNT=8 # REPLICA_COUNT=1 # ES_URL=http://localhost:9200 # while [["$(curl -s -o /dev/null -w '%{http_code}\n' $ES_URL)" != "200"]]; do sleep 1; done # curl -XPUT "$ES_URL/_template/$TEMPLATE_NAME" -H 'Content-Type: application/json' -d'{"index_patterns":['\""$INDEX_PATTERN"\"'],"settings":{"number_of_shards":'$SHARD_COUNT',"number_of_replicas":'$REPLICA_COUNT'}}' sysctlInitContainer: enabled: true keystore: [] # Deprecated # please use the above podSecurityContext.fsGroup instead fsGroup: "1000" ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 17, 2021, 11:04am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/12 "2021-05-17T11:04:48Z")
I honestly don't know anything about running Elasticsearch in such an env... I just have a few comments. - I'd use the 7.12.1 instead of 7.6.1 which is quite old. And the `node.remote_cluster_client` has been added I think in 7.7. - I don't see anywhere the `node.remote_cluster_client` setting so I'm not sure where this is coming from. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 14, 2021, 11:05am UTC](https://discuss.elastic.co/t/elastic-application-is-down-because-of-unknown-node-remote-cluster-client/272484/13 "2021-06-14T11:05:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.