Labels and Annotation are not applied at the pod level

Hi,

I'm trying to do a poc with datadog to monitor elasticsearch.
One of the solution is to setting annotations and labels on pods.
So I tried to do that, using the podTemplate field in the CRD but it doesn't update the stateful set.
I have to following manifest:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  annotations:
    common.k8s.elastic.co/controller-version: 1.3.0
    elasticsearch.k8s.elastic.co/cluster-uuid: R-x_d2qVQO6VkfjLvauX8w
    meta.helm.sh/release-name: eck-elastic-alumni-connect
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2020-11-10T06:23:04Z"
  generation: 8
  labels:
    app.kubernetes.io/managed-by: Helm
  name: alumni-connect
  namespace: default
  resourceVersion: "222736386"
  selfLink: /apis/elasticsearch.k8s.elastic.co/v1/namespaces/default/elasticsearches/alumni-connect
  uid: ec916938-0802-489a-bbb1-9cee580d9c59
spec:
  auth:
    fileRealm:
    - secretName: elk-users-secret
    roles:
    - secretName: elk-roles-secret
  http:
    service:
      metadata:
        creationTimestamp: null
      spec: {}
    tls:
      certificate: {}
      selfSignedCertificate:
        disabled: true
  image: gcr.io/hivebrite/elasticsearch7:481d0cda7
  nodeSets:
  - config:
      node.data: true
      node.master: true
    count: 3
    name: all
    podTemplate:
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: workloadType
                  operator: In
                  values:
                  - elasticsearch3
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: workloadName
                    operator: In
                    values:
                    - alumni-connect
                  - key: workloadScope
                    operator: In
                    values:
                    - alumni_connect
                  - key: workloadStack
                    operator: In
                    values:
                    - alumni_connect
                topologyKey: kubernetes.io/hostname
              weight: 100
        containers:
        - env:
          - name: ES_JAVA_OPTS
            value: -Xms4028m -Xmx4028m
          name: elasticsearch
          resources:
            limits:
              memory: 5000Mi
            requests:
              cpu: 1000m
              memory: 5000Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - SETPCAP
              - MKNOD
              - AUDIT_WRITE
              - CHOWN
              - NET_RAW
              - DAC_OVERRIDE
              - FOWNER
              - FSETID
              - KILL
              - SETGID
              - SETUID
              - NET_BIND_SERVICE
              - SYS_CHROOT
              - SETFCAP
            runAsUser: 1000
        initContainers:
        - command:
          - sh
          - -c
          - sysctl -w vm.max_map_count=262144
          name: sysctl
          securityContext:
            privileged: true
        metadata:
          annotations:
            ad.datadoghq.com/elastic.check_names: '["elastic"]'
            ad.datadoghq.com/elastic.init_configs: '[{}]'
            ad.datadoghq.com/elastic.instances: |
              [
                {
                  "url": "%%host%%",
                  "username": "elastic"
                  "password":"ENC[kubernetes_default_alumni-connect-es-elastic-user_elastic]",
                  "auth_type": "basic",
                  "cluster_stats": true
                  "tags": {
                    "env": "wip",
                    "service": "alumni-connect",
                    "version": "7.9.3"
                  }
                }
              ]
          labels:
            clusterName: wip
            region: europe-west1
            workloadName: alumni-connect
            workloadScope: alumni_connect
            workloadStack: alumni_connect
        tolerations:
        - key: workloadType
          operator: Equal
          value: elasticsearch3
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 500Gi
        storageClassName: ssd
  secureSettings:
  - secretName: alumni-connect-eck-gcp
  transport:
    service:
      metadata:
        creationTimestamp: null
      spec: {}
  updateStrategy:
    changeBudget:
      maxSurge: 1
      maxUnavailable: 1
  version: 7.9.3
status:
  availableNodes: 3
  health: green
  phase: Ready
  version: 7.9.3

and my statefulset manifest is:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2020-11-10T06:23:09Z"
  generation: 7
  labels:
    common.k8s.elastic.co/template-hash: "90437220"
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: alumni-connect
    elasticsearch.k8s.elastic.co/statefulset-name: alumni-connect-es-all
  name: alumni-connect-es-all
  namespace: default
  ownerReferences:
  - apiVersion: elasticsearch.k8s.elastic.co/v1
    blockOwnerDeletion: true
    controller: true
    kind: Elasticsearch
    name: alumni-connect
    uid: ec916938-0802-489a-bbb1-9cee580d9c59
  resourceVersion: "222725688"
  selfLink: /apis/apps/v1/namespaces/default/statefulsets/alumni-connect-es-all
  uid: e73aeec5-572a-40d8-9a8f-68d6bd29970a
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      common.k8s.elastic.co/type: elasticsearch
      elasticsearch.k8s.elastic.co/cluster-name: alumni-connect
      elasticsearch.k8s.elastic.co/statefulset-name: alumni-connect-es-all
  serviceName: alumni-connect-es-all
  template:
    metadata:
      annotations:
        co.elastic.logs/module: elasticsearch
      creationTimestamp: null
      labels:
        common.k8s.elastic.co/type: elasticsearch
        elasticsearch.k8s.elastic.co/cluster-name: alumni-connect
        elasticsearch.k8s.elastic.co/config-hash: "2296861737"
        elasticsearch.k8s.elastic.co/http-scheme: http
        elasticsearch.k8s.elastic.co/node-data: "true"
        elasticsearch.k8s.elastic.co/node-ingest: "true"
        elasticsearch.k8s.elastic.co/node-master: "true"
        elasticsearch.k8s.elastic.co/node-ml: "true"
        elasticsearch.k8s.elastic.co/node-transform: "true"
        elasticsearch.k8s.elastic.co/secure-settings-hash: 5477e82f85823f08a2712e4dad1f964bf1d28b419e8b00466cd2d3b9
        elasticsearch.k8s.elastic.co/statefulset-name: alumni-connect-es-all
        elasticsearch.k8s.elastic.co/version: 7.9.3
    ....

My cluster seems ok:

kubectl get elasticsearches.elasticsearch.k8s.elastic.co alumni-connect
NAME             HEALTH   NODES   VERSION   PHASE   AGE
alumni-connect   green    3       7.9.3     Ready   79d

The version of the operator is: 1.3.0

Someone have an idea why this conf is not working ? or How to apply labels and annotations at the pod labels

Best regards,
Damien

You have added the annotations and labels under podTemplate.spec whereas they should be added under podTemplate.metadata. For example:

---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: hulk
spec:
  version: 7.10.0
  nodeSets:
    - name: default
      count: 3
      podTemplate:
        metadata:
          annotations:
            my-annotation: "annotation-value"
          labels:
            my-label: "label-value"

Good catch, it works :tada:
Thank you for the help

Regards,
Damien