Failed to setup automated snapshot with elastic operator

Elastic Stack Elasticsearch
, ,
1

I'm launching 3 elastic nodes using elastic operator and i tried to set up automated snapshots for these instances.

I used elastic operator 2.5.0 on Kubernetes version v1.23.11-gke.300 and used Elasticsearch version 8.5.3 and Kibana version 8.5.3

I followed this Create automated snapshots | Elastic Cloud on Kubernetes [2.5] | Elastic

I minified the Json of the service account key and created a file called "gcs.client.default.credentials_file" with no file extension and added this file to Kubernetes secrets.

This is the command i used to build the k8s secret
kubectl create secret generic gcs-credentials --from-file=gcs.client.default.credentials_file -n elastic-system

This is the configrations yaml file:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-cluster
spec:
  version: 8.5.3
  secureSettings:
  - secretName: gcs-credentials
  nodeSets:
  - name: node
    count: 3
    config:
      node.roles: ['master', 'data', 'ingest', 'ml', 'remote_cluster_client', 'transform']
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install analysis-icu
            bin/elasticsearch-plugin install analysis-kuromoji
            bin/elasticsearch-plugin install analysis-smartcn
            bin/elasticsearch-plugin install analysis-stempel
            bin/elasticsearch-plugin install analysis-nori
            bin/elasticsearch-plugin install analysis-phonetic
            bin/elasticsearch-plugin install analysis-ukrainian

        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms1g -Xmx1g"

          resources:
            requests:
              cpu: 1000m
              memory: 2Gi
            limits:
              cpu: 2000m
              memory: 4Gi
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "standard"
        resources:
          requests:
            storage: 100Gi
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 8.5.3
  count: 1
  elasticsearchRef:
    name: elasticsearch-cluster
  config:
    xpack.reporting.roles.enabled: false
    xpack.reporting.kibanaServer.hostname: localhost

And added the secureSettings.secretName field to the spec of the elastic cluster and added the secret name to it which was "gcs-credentials"

But I get this error on the logs which saying:
failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]

`{"@timestamp":"2022-12-26T18:45:40.037Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","` [`service.name`](https://service.name/)`":"ES_ECS","event.dataset":"elasticsearch.server","` [`process.thread.name`](https://process.thread.name/)`":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","` [`elasticsearch.node.name`](https://elasticsearch.node.name/)`":"elasticsearch-cluster-es-node-1","` [`elasticsearch.cluster.name`](https://elasticsearch.cluster.name/)`":"elasticsearch-cluster","error.type":"java.lang.IllegalStateException","error.message":"failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]","error.stack_trace":"java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.loadPlugin(` [`PluginsService.java:607`](https://pluginsservice.java:607/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.loadBundle(` [`PluginsService.java:482`](https://pluginsservice.java:482/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.loadBundles(` [`PluginsService.java:290`](https://pluginsservice.java:290/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.<init>(` [`PluginsService.java:159`](https://pluginsservice.java:159/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(` [`PluginsService.java:634`](https://pluginsservice.java:634/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(` [`Node.java:406`](https://node.java:406/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(` [`Node.java:316`](https://node.java:316/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(` [`Elasticsearch.java:214`](https://elasticsearch.java:214/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(` [`Elasticsearch.java:214`](https://elasticsearch.java:214/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.main(` [`Elasticsearch.java:67`](https://elasticsearch.java:67/)`)\nCaused by: java.lang.reflect.InvocationTargetException\n\tat java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(` [`DirectConstructorHandleAccessor.java:79`](https://directconstructorhandleaccessor.java:79/)`)\n\tat java.base/java.lang.reflect.Constructor.newInstanceWithCaller(` [`Constructor.java:500`](https://constructor.java:500/)`)\n\tat java.base/java.lang.reflect.Constructor.newInstance(` [`Constructor.java:484`](https://constructor.java:484/)`)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.loadPlugin(` [`PluginsService.java:600`](https://pluginsservice.java:600/)`)\n\t... 9 more\nCaused by: java.lang.IllegalArgumentException: failed to load GCS client credentials from [gcs.client.default.credentials_file]\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(` [`GoogleCloudStorageClientSettings.java:265`](https://googlecloudstorageclientsettings.java:265/)`)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.getClientSettings(` [`GoogleCloudStorageClientSettings.java:221`](https://googlecloudstorageclientsettings.java:221/)`)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.load(` [`GoogleCloudStorageClientSettings.java:209`](https://googlecloudstorageclientsettings.java:209/)`)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.reload(` [`GoogleCloudStoragePlugin.java:88`](https://googlecloudstorageplugin.java:88/)`)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.<init>(` [`GoogleCloudStoragePlugin.java:36`](https://googlecloudstorageplugin.java:36/)`)\n\tat java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(` [`DirectConstructorHandleAccessor.java:67`](https://directconstructorhandleaccessor.java:67/)`)\n\t... 12 more\nCaused by: java.io.IOException: Invalid PKCS#8 data.\n\tat com.google.auth.oauth2.ServiceAccountCredentials.privateKeyFromPkcs8(` [`ServiceAccountCredentials.java:496`](https://serviceaccountcredentials.java:496/)`)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromPkcs8(` [`ServiceAccountCredentials.java:474`](https://serviceaccountcredentials.java:474/)`)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromJson(` [`ServiceAccountCredentials.java:212`](https://serviceaccountcredentials.java:212/)`)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(` [`ServiceAccountCredentials.java:548`](https://serviceaccountcredentials.java:548/)`)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(` [`ServiceAccountCredentials.java:520`](https://serviceaccountcredentials.java:520/)`)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.lambda$loadCredential$13(` [`GoogleCloudStorageClientSettings.java:257`](https://googlecloudstorageclientsettings.java:257/)`)\n\tat java.base/java.security.AccessController.doPrivileged(` [`AccessController.java:569`](https://accesscontroller.java:569/)`)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(` [`SocketAccess.java:33`](https://socketaccess.java:33/)`)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(` [`GoogleCloudStorageClientSettings.java:256`](https://googlecloudstorageclientsettings.java:256/)`)\n\t... 17 more\n"}`

`ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elasticsearch-cluster.log`
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.