# Gcs-repository-creation-issue for elasticsearch on gke **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [September 29, 2019, 6:30am UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519 "2019-09-29T06:30:55Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Kumar\_Saurabh\_Srivas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kumar_saurabh_srivas/32/55080_2.png) [@Kumar\_Saurabh\_Srivas](https://discuss.elastic.co/u/Kumar_Saurabh_Srivas)\ **Post date:** [September 29, 2019, 6:30am UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/1 "2019-09-29T06:30:55Z")
We are setting up an elasticsearch cluster on GKE with the following format: 1. Master nodes as kubernetes deployments 2. Client nodes as kubernetes deployments with HPA 3. Data nodes as stateful sets with PVs We are able to set up the cluster well. But then we are struggling in configuring the snapshot backup mechanism. Essentially, we are following [this](https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-gcs-usage.html) guide. We are able to follow this upto the step of getting the secret json key. Afterwards, we are not sure how to add this to the elasticsearch keystore and proceed further. We are really stuck on this for quite some and the documentation have not been great. All docs mention that add this json key to elasticsearch.keystore but we don't know how to do that. The json file is on our local shell while the keystore is on es pods. Also, we have created a custom dockerfile to install gcs plugin. Really looking for some help here. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 29, 2019, 6:55am UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/2 "2019-09-29T06:55:50Z")
Are you using [Elastic Cloud on Kubernetes](https://www.elastic.co/elasticsearch-kubernetes)? If so, you can [get secrets into the keystore from the Kubernetes secret store](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-snapshot.html#k8s-secure-settings). ---
**Author:** ![Kumar\_Saurabh\_Srivas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kumar_saurabh_srivas/32/55080_2.png) [@Kumar\_Saurabh\_Srivas](https://discuss.elastic.co/u/Kumar_Saurabh_Srivas)\ **Post date:** [September 30, 2019, 2:59am UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/3 "2019-09-30T02:59:26Z")
We aren't using elastic cloud on Kubernetes. We have created a Kubernetes cluster using Elasticsearch docker image. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 30, 2019, 6:51am UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/4 "2019-09-30T06:51:17Z")
The orchestration of an Elasticsearch cluster is not simple, and it's easy to get it wrong in ways that occasionally lose data. I recommend using the official operator rather than trying to develop your own orchestration. If you insist on using your own images, you should use the [`elasticsearch-keystore` command to add any secrets to the keystore](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/secure-settings.html#secure-settings). ---
**Author:** ![Kumar\_Saurabh\_Srivas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kumar_saurabh_srivas/32/55080_2.png) [@Kumar\_Saurabh\_Srivas](https://discuss.elastic.co/u/Kumar_Saurabh_Srivas)\ **Post date:** [September 30, 2019, 8:30am UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/5 "2019-09-30T08:30:29Z")
One of the core reasons why we did not go ahead with the official version is that we were not sure if we can configure it correctly to operate at loads such as 100k RPS for both reads and writes. It seems that we should really try this out. I would love to hear your suggestions about what sort of configuration should we have. In our existing ES cluster, we have about 500 GB of data and about 100k RPS. We are planning to use 8 vCPU 32 GB machines for our Kubernetes ES cluster so that we can have heap size of about 14-15 GB. Can you suggest some configuration tips / suggestions based on your experience with this operator. Also, how does the operator take care of autoscaling, especially of data and client nodes here? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 30, 2019, 11:17am UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/6 "2019-09-30T11:17:57Z")
> [@Kumar\_Saurabh\_Srivas](#): > > One of the core reasons why we did not go ahead with the official version is that we were not sure if we can configure it correctly to operate at loads such as 100k RPS for both reads and writes. I do not think the orchestration mechanism should have any impact on performance. You should get the same cluster however it's orchestrated. > [@Kumar\_Saurabh\_Srivas](#): > > Can you suggest some configuration tips / suggestions based on your experience with this operator [Benchmark](https://esrally.readthedocs.io) your setup with a realistic workload. That's the only way you can truly validate its performance characteristics. Our [public benchmarks](https://elasticsearch-benchmarks.elastic.co/#tracks/http-logs/nightly/default/30d) show performance on some workloads in excess of 100k per second on a three-node benchmarking cluster, but performance is very dependent on your workload and hardware so you must perform your own experiments. > [@Kumar\_Saurabh\_Srivas](#): > > how does the operator take care of autoscaling, especially of data and client nodes here I don't think there's any auto-scaling yet. It doesn't seem necessary in such a small cluster. ---
**Author:** ![Kumar\_Saurabh\_Srivas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kumar_saurabh_srivas/32/55080_2.png) [@Kumar\_Saurabh\_Srivas](https://discuss.elastic.co/u/Kumar_Saurabh_Srivas)\ **Post date:** [September 30, 2019, 2:09pm UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/7 "2019-09-30T14:09:14Z")
I tried crating a cluster. But some of my pods are being OOM killed which is strange given that I am using 8vCPU 30 GB machines. Ideally, I would have expected this to work well given the resources I am using. Below is my elasticsearch.yaml: kind: StorageClass apiVersion: [storage.k8s.io/v1](http://storage.k8s.io/v1) metadata: name: local-storage namespace: elastic-system provisioner: [kubernetes.io/no-provisioner](http://kubernetes.io/no-provisioner) volumeBindingMode: WaitForFirstConsumer * * * apiVersion: [elasticsearch.k8s.elastic.co/v1alpha1](http://elasticsearch.k8s.elastic.co/v1alpha1) kind: Elasticsearch metadata: name: es-cluster namespace: elastic-system spec: version: 7.2.0 nodes: # 3 dedicated master nodes - nodeCount: 3 config: node.master: true node.data: false node.ingest: false cluster.remote.connect: false podTemplate: spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: [elasticsearch.k8s.elastic.co/cluster-name:](http://elasticsearch.k8s.elastic.co/cluster-name:) es-cluster topologyKey: [kubernetes.io/hostname](http://kubernetes.io/hostname) nodeSelector: [cloud.google.com/gke-nodepool:](http://cloud.google.com/gke-nodepool:) es-pool initContainers: - name: init-sysctl image: busybox:1.27.2 command: - sysctl - -w - vm.max\_map\_count=2621441 securityContext: privileged: true - name: install-plugins command: - sh - -c - | bin/elasticsearch-plugin install --batch repository-gcs containers: - name: elasticsearch env: - name: ES\_JAVA\_OPTS value: -Xms5g -Xmx5g - name: NSS\_SDB\_USE\_CACHE value: "no" resources: requests: memory: 2Gi limits: cpu: 4 memory: 5Gi # 3 coordinating nodes - nodeCount: 3 config: node.master: false node.data: false node.ingest: false podTemplate: spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: [elasticsearch.k8s.elastic.co/cluster-name:](http://elasticsearch.k8s.elastic.co/cluster-name:) es-cluster topologyKey: [kubernetes.io/hostname](http://kubernetes.io/hostname) nodeSelector: [cloud.google.com/gke-nodepool:](http://cloud.google.com/gke-nodepool:) es-pool initContainers: - name: init-sysctl image: busybox:1.27.2 command: - sysctl - -w - vm.max\_map\_count=2621441 securityContext: privileged: true - name: install-plugins command: - sh - -c - | bin/elasticsearch-plugin install --batch repository-gcs containers: - name: elasticsearch env: - name: ES\_JAVA\_OPTS value: -Xms7g -Xmx7g - name: NSS\_SDB\_USE\_CACHE value: "no" resources: requests: memory: 5Gi limits: cpu: 4 memory: 6Gi # 4 data nodes - nodeCount: 4 config: node.master: false node.data: true node.ingest: false podTemplate: spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: [elasticsearch.k8s.elastic.co/cluster-name:](http://elasticsearch.k8s.elastic.co/cluster-name:) es-cluster topologyKey: [kubernetes.io/hostname](http://kubernetes.io/hostname) nodeSelector: [cloud.google.com/gke-nodepool:](http://cloud.google.com/gke-nodepool:) es-pool initContainers: - name: init-sysctl image: busybox:1.27.2 command: - sysctl - -w - vm.max\_map\_count=2621441 securityContext: privileged: true - name: install-plugins command: - sh - -c - | bin/elasticsearch-plugin install --batch repository-gcs containers: - name: elasticsearch env: - name: ES\_JAVA\_OPTS value: -Xms5g -Xmx5g - name: NSS\_SDB\_USE\_CACHE value: "no" resources: requests: memory: 5Gi limits: cpu: 4 memory: 5Gi - volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 200Gi storageClassName: local-storage updateStrategy: changeBudget: maxSurge: 1 maxUnavailable: 0 secureSettings: secretName: gcs-credentials http: service: spec: type: ClusterIP ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 30, 2019, 2:25pm UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/8 "2019-09-30T14:25:20Z")
Please use the `` button to format any YAML you are sharing properly. YAML is whitespace-sensitive and if you don't format it properly then it's quite meaningless. You should read [the guide to setting the heap size](https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html), in particular: > Set `Xmx` and `Xms` to no more than 50% of your physical RAM. Elasticsearch requires memory for purposes other than the JVM heap and it is important to leave space for this... Here "physical RAM" means "RAM allocated to the container". Your coordinating nodes have a 7GB heap on a 6GB container which is completely hopeless, and the other containers have heap size equal to container RAM which is still off by a factor of 2. ---
**Author:** ![Kumar\_Saurabh\_Srivas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kumar_saurabh_srivas/32/55080_2.png) [@Kumar\_Saurabh\_Srivas](https://discuss.elastic.co/u/Kumar_Saurabh_Srivas)\ **Post date:** [September 30, 2019, 3:14pm UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/9 "2019-09-30T15:14:12Z")
Sorry for the bad format. But thanks a lot. The cluster is up and running. I am going ahead with configuring snapshots now. ---
**Author:** ![Kumar\_Saurabh\_Srivas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kumar_saurabh_srivas/32/55080_2.png) [@Kumar\_Saurabh\_Srivas](https://discuss.elastic.co/u/Kumar_Saurabh_Srivas)\ **Post date:** [September 30, 2019, 5:42pm UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/10 "2019-09-30T17:42:04Z")
I have updated it and the cluster is running fine. I have created a clusterIP service: ```auto NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elastic-webhook-service ClusterIP 10.64.4.214 443/TCP 4h10m service/es-cluster-es-http ClusterIP 10.64.7.198 9200/TCP 163m ``` But when I ssh into a node of the Kubernetes cluster (in another node pool which is not running the ES cluster) and I try curl commands, I get no reply from server. curl: (52) Empty reply from server Any idea what is happening? Not sure if my cluster is running. How do I create indexes and insert data? curl -X GET "10.64.7.198:9200/\_cluster/health?pretty": curl: (52) Empty reply from server kubernetes -n elastic-system get elasticsearch: ```auto NAME HEALTH NODES VERSION PHASE AGE es-cluster green 10 7.2.0 Operational 2h ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 30, 2019, 5:54pm UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/11 "2019-09-30T17:54:23Z")
That sounds like possibly a network config issue, but I'm not the best person to ask about this. I've moved this post over to the ECK forum and hopefully someone else can help with the details here. ---
**Author:** ![Anya\_Sabo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anya_sabo/32/49903_2.png) [@Anya\_Sabo](https://discuss.elastic.co/u/Anya_Sabo)\ **Post date:** [September 30, 2019, 6:10pm UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/12 "2019-09-30T18:10:11Z")
Hello, please see the docs here: [https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-accessing-elastic-services.html](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-accessing-elastic-services.html) for how to access it. In your case it looks like you will need to specify https and authenticate to connect to the ES cluster. If you run into any issues please let us know, we try to document the common ones people run into. ---
**Author:** ![Kumar\_Saurabh\_Srivas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kumar_saurabh_srivas/32/55080_2.png) [@Kumar\_Saurabh\_Srivas](https://discuss.elastic.co/u/Kumar_Saurabh_Srivas)\ **Post date:** [October 1, 2019, 4:14am UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/13 "2019-10-01T04:14:09Z")
Yeah. Got it working. Thanks a lot. This entire thread has been super useful. My cluster is up and running. I just have one concern here that there is no auto scaling. In case my pods are reaching CPU limits, I don't have something like a Kubernetes Horizontal Pod Autoscaler to automatically schedule more pods. I will have to monitor the cluster myself for each of the possible bottlenecks and then scale my cluster manually. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 29, 2019, 4:14am UTC](https://discuss.elastic.co/t/gcs-repository-creation-issue-for-elasticsearch-on-gke/201519/14 "2019-10-29T04:14:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.