Elasticsearch trial license downgraded to basic after pod recreation on Kubernetes

I have deployed a ES stack on Kubernetes with the operator (ECK). Just ES + Kibana, and it is Ok.

I’m trying to test some enterprise functions, and activate a trial licence, but elasticsearch reverts to a basic license after each configuration change or pod delete/create

What I did in summary

  • Create a yaml file for the operator to activate the license
  • kubectl apply to activate it
  • Create the Elasticsearch service (Pod) named es-prod

What I see

  • The operator activates the license.
  • I see a ConfigMap elastic-licensing with a valid license
  • Elasticsearch starts, and has a valid trial license. Checked on the created secret named es-prod-es-license, and with an API call on /_license
  • I see a lot of creations / changes on the secret named es-prod-es-license (the uid changes)
  • If I change ES conf or delete the Pod (which is then recreated), ES starts with the trial license, and replaces it by the basic license

Some technical infos

Operator

Namespace: app-stats-operator-es-prod

Conf:

Source: eck-operator/templates/service-account.yaml

apiVersion: v1

kind: ServiceAccount

automountServiceAccountToken: true

metadata:

name: elastic-stats-prod-operator

namespace: app-stats-operator-es-prod

labels:

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"




Source: eck-operator/templates/webhook.yaml

apiVersion: v1

kind: Secret

metadata:

name: elastic-webhook-server-cert

namespace: app-stats-operator-es-prod

labels:

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"




Source: eck-operator/templates/configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

name: elastic-stats-prod-operator

namespace: app-stats-operator-es-prod

labels:

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"

data:

eck.yaml: |-

log-verbosity: 0

metrics-port: 0

metrics-secure: false

container-registry: docker.elastic.co

max-concurrent-reconciles: 3

ca-cert-validity: 8760h

ca-cert-rotate-before: 24h

cert-validity: 8760h

cert-rotate-before: 24h

disable-config-watch: false

exposed-node-labels: [topology.kubernetes.io/.,failure-domain.beta.kubernetes.io/.]

set-default-security-context: auto-detect

kube-client-timeout: 60s

elasticsearch-client-timeout: 180s

disable-telemetry: false

distribution-channel: all-in-one

validate-storage-class: true

enable-webhook: true

webhook-name: elastic-webhook-prod.k8s.elastic.co

webhook-port: 9443

operator-namespace: app-stats-operator-es-prod

namespaces: app-stats-es-prod,app-stats-kibana-prod,app-stats-logstash-prod,app-stats-operator-es-prod

enable-leader-election: true

elasticsearch-observation-interval: 10s

ubi-only: false




Source: eck-operator/templates/cluster-roles.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

name: elastic-stats-prod-operator

labels:

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"

rules:

apiGroups:

- "authorization.k8s.io"

resources:

- subjectaccessreviews

verbs:

- create

apiGroups:

- coordination.k8s.io

resources:

- leases

verbs:

- create

apiGroups:

- coordination.k8s.io

resources:

- leases

resourceNames:

- elastic-operator-leader

verbs:

- get

- watch

- update

apiGroups:

- ""

resources:

- endpoints

verbs:

- get

- list

- watch

apiGroups:

- ""

resources:

- pods

- events

- persistentvolumeclaims

- secrets

- services

- configmaps

verbs:

- get

- list

- watch

- create

- update

- patch

- delete

apiGroups:

- apps

resources:

- deployments

- statefulsets

- daemonsets

verbs:

- get

- list

- watch

- create

- update

- patch

- delete

apiGroups:

- policy

resources:

- poddisruptionbudgets

verbs:

- get

- list

- watch

- create

- update

- patch

- delete

apiGroups:

- elasticsearch.k8s.elastic.co

resources:

- elasticsearches

- elasticsearches/status

- elasticsearches/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- autoscaling.k8s.elastic.co

resources:

- elasticsearchautoscalers

- elasticsearchautoscalers/status

- elasticsearchautoscalers/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- kibana.k8s.elastic.co

resources:

- kibanas

- kibanas/status

- kibanas/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- apm.k8s.elastic.co

resources:

- apmservers

- apmservers/status

- apmservers/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- enterprisesearch.k8s.elastic.co

resources:

- enterprisesearches

- enterprisesearches/status

- enterprisesearches/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- beat.k8s.elastic.co

resources:

- beats

- beats/status

- beats/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- agent.k8s.elastic.co

resources:

- agents

- agents/status

- agents/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- maps.k8s.elastic.co

resources:

- elasticmapsservers

- elasticmapsservers/status

- elasticmapsservers/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- stackconfigpolicy.k8s.elastic.co

resources:

- stackconfigpolicies

- stackconfigpolicies/status

- stackconfigpolicies/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- logstash.k8s.elastic.co

resources:

- logstashes

- logstashes/status

- logstashes/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP

verbs:

- get

- list

- watch

- create

- update

- patch

apiGroups:

- storage.k8s.io

resources:

- storageclasses

verbs:

- get

- list

- watch

apiGroups:

- admissionregistration.k8s.io

resources:

- validatingwebhookconfigurations

verbs:

- get

- list

- watch

- create

- update

- patch

- delete

apiGroups:

- ""

resources:

- nodes

verbs:

- get

- list

- watch




Source: eck-operator/templates/cluster-roles.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

name: "elastic-stats-prod-operator-view"

labels:

rbac.authorization.k8s.io/aggregate-to-view: "true"

rbac.authorization.k8s.io/aggregate-to-edit: "true"

rbac.authorization.k8s.io/aggregate-to-admin: "true"

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"

rules:

apiGroups: ["elasticsearch.k8s.elastic.co"]

resources: ["elasticsearches"]

verbs: ["get", "list", "watch"]

apiGroups: ["autoscaling.k8s.elastic.co"]

resources: ["elasticsearchautoscalers"]

verbs: ["get", "list", "watch"]

apiGroups: ["apm.k8s.elastic.co"]

resources: ["apmservers"]

verbs: ["get", "list", "watch"]

apiGroups: ["kibana.k8s.elastic.co"]

resources: ["kibanas"]

verbs: ["get", "list", "watch"]

apiGroups: ["enterprisesearch.k8s.elastic.co"]

resources: ["enterprisesearches"]

verbs: ["get", "list", "watch"]

apiGroups: ["beat.k8s.elastic.co"]

resources: ["beats"]

verbs: ["get", "list", "watch"]

apiGroups: ["agent.k8s.elastic.co"]

resources: ["agents"]

verbs: ["get", "list", "watch"]

apiGroups: ["maps.k8s.elastic.co"]

resources: ["elasticmapsservers"]

verbs: ["get", "list", "watch"]

apiGroups: ["stackconfigpolicy.k8s.elastic.co"]

resources: ["stackconfigpolicies"]

verbs: ["get", "list", "watch"]

apiGroups: ["logstash.k8s.elastic.co"]

resources: ["logstashes"]

verbs: ["get", "list", "watch"]




Source: eck-operator/templates/cluster-roles.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

name: "elastic-stats-prod-operator-edit"

labels:

rbac.authorization.k8s.io/aggregate-to-edit: "true"

rbac.authorization.k8s.io/aggregate-to-admin: "true"

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"

rules:

apiGroups: ["elasticsearch.k8s.elastic.co"]

resources: ["elasticsearches"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]

apiGroups: ["autoscaling.k8s.elastic.co"]

resources: ["elasticsearchautoscalers"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]

apiGroups: ["apm.k8s.elastic.co"]

resources: ["apmservers"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]

apiGroups: ["kibana.k8s.elastic.co"]

resources: ["kibanas"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]

apiGroups: ["enterprisesearch.k8s.elastic.co"]

resources: ["enterprisesearches"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]

apiGroups: ["beat.k8s.elastic.co"]

resources: ["beats"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]

apiGroups: ["agent.k8s.elastic.co"]

resources: ["agents"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]

apiGroups: ["maps.k8s.elastic.co"]

resources: ["elasticmapsservers"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]

apiGroups: ["stackconfigpolicy.k8s.elastic.co"]

resources: ["stackconfigpolicies"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]

apiGroups: ["logstash.k8s.elastic.co"]

resources: ["logstashes"]

verbs: ["create", "delete", "deletecollection", "patch", "update"]




Source: eck-operator/templates/role-bindings.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

name: elastic-stats-prod-operator

labels:

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: elastic-stats-prod-operator

subjects:

kind: ServiceAccount

name: elastic-stats-prod-operator

namespace: app-stats-operator-es-prod




Source: eck-operator/templates/webhook.yaml

apiVersion: v1

kind: Service

metadata:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

labels:

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"

spec:

ports:

- name: https

port: 443

targetPort: 9443

selector:

control-plane: elastic-stats-prod-operator




Source: eck-operator/templates/statefulset.yaml

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: elastic-stats-prod-operator

namespace: app-stats-operator-es-prod

labels:

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"

spec:

selector:

matchLabels:

control-plane: elastic-stats-prod-operator

serviceName: elastic-stats-prod-operator

replicas: 1

template:

metadata:

annotations:

# Rename the fields "error" to "error.message" and "source" to "event.source"

# This is to avoid a conflict with the ECS "error" and "source" documents.

"co.elastic.logs/raw": "[{"type":"container","json.keys_under_root":true,"paths":["/var/log/containers/*${data.kubernetes.container.id}.log"],"processors":[{"convert":{"mode":"rename","ignore_missing":true,"fields":[{"from":"error","to":"_error"}]}},{"convert":{"mode":"rename","ignore_missing":true,"fields":[{"from":"_error","to":"error.message"}]}},{"convert":{"mode":"rename","ignore_missing":true,"fields":[{"from":"source","to":"_source"}]}},{"convert":{"mode":"rename","ignore_missing":true,"fields":[{"from":"_source","to":"event.source"}]}}]}]"

"checksum/config": 3fdaff4a979089797cb469806c19162f44297428bf811896931864c528196029

labels:

control-plane: elastic-stats-prod-operator

spec:

terminationGracePeriodSeconds: 10

serviceAccountName: elastic-stats-prod-operator

automountServiceAccountToken: true

securityContext:

runAsNonRoot: true

containers:

- image: "inasoft-harbor.jetpulp.hosting/hub/elastic/eck-operator:3.2.0"

imagePullPolicy: IfNotPresent

name: manager

args:

- "manager"

- "--config=/conf/eck.yaml"

securityContext:

allowPrivilegeEscalation: false

capabilities:

drop:

- ALL

readOnlyRootFilesystem: true

runAsNonRoot: true

env:

- name: OPERATOR_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

- name: POD_IP

valueFrom:

fieldRef:

fieldPath: status.podIP

- name: WEBHOOK_SECRET

value: elastic-webhook-server-cert

resources:

limits:

cpu: 1

memory: 1Gi

requests:

cpu: 100m

memory: 150Mi

ports:

- containerPort: 9443

name: https-webhook

protocol: TCP

volumeMounts:

- mountPath: "/conf"

name: conf

readOnly: true

- mountPath: /tmp/k8s-webhook-server/serving-certs

name: cert

readOnly: true

volumes:

- name: conf

configMap:

name: elastic-stats-prod-operator

- name: cert

secret:

defaultMode: 420

secretName: elastic-webhook-server-cert

imagePullSecrets:

- name: hub-proxy-pull-secret




Source: eck-operator/templates/webhook.yaml

apiVersion: admissionregistration.k8s.io/v1

kind: ValidatingWebhookConfiguration

metadata:

name: elastic-webhook-prod.k8s.elastic.co

labels:

control-plane: elastic-stats-prod-operator

app.kubernetes.io/version: "3.2.0"

webhooks:

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-agent-k8s-elastic-co-v1alpha1-agent

failurePolicy: Ignore

name: elastic-agent-validation-v1alpha1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- agent.k8s.elastic.co

apiVersions:

- v1alpha1

operations:

- CREATE

- UPDATE

resources:

- agents

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-apm-k8s-elastic-co-v1-apmserver

failurePolicy: Ignore

name: elastic-apm-validation-v1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- apm.k8s.elastic.co

apiVersions:

- v1

operations:

- CREATE

- UPDATE

resources:

- apmservers

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-apm-k8s-elastic-co-v1beta1-apmserver

failurePolicy: Ignore

name: elastic-apm-validation-v1beta1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- apm.k8s.elastic.co

apiVersions:

- v1beta1

operations:

- CREATE

- UPDATE

resources:

- apmservers

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-beat-k8s-elastic-co-v1beta1-beat

failurePolicy: Ignore

name: elastic-beat-validation-v1beta1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- beat.k8s.elastic.co

apiVersions:

- v1beta1

operations:

- CREATE

- UPDATE

resources:

- beats

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-enterprisesearch-k8s-elastic-co-v1-enterprisesearch

failurePolicy: Ignore

name: elastic-ent-validation-v1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- enterprisesearch.k8s.elastic.co

apiVersions:

- v1

operations:

- CREATE

- UPDATE

resources:

- enterprisesearches

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-enterprisesearch-k8s-elastic-co-v1beta1-enterprisesearch

failurePolicy: Ignore

name: elastic-ent-validation-v1beta1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- enterprisesearch.k8s.elastic.co

apiVersions:

- v1beta1

operations:

- CREATE

- UPDATE

resources:

- enterprisesearches

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-elasticsearch-k8s-elastic-co-v1-elasticsearch

failurePolicy: Ignore

name: elastic-es-validation-v1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- elasticsearch.k8s.elastic.co

apiVersions:

- v1

operations:

- CREATE

- UPDATE

resources:

- elasticsearches

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-elasticsearch-k8s-elastic-co-v1beta1-elasticsearch

failurePolicy: Ignore

name: elastic-es-validation-v1beta1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- elasticsearch.k8s.elastic.co

apiVersions:

- v1beta1

operations:

- CREATE

- UPDATE

resources:

- elasticsearches

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-ems-k8s-elastic-co-v1alpha1-mapsservers

failurePolicy: Ignore

name: elastic-ems-validation-v1alpha1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- maps.k8s.elastic.co

apiVersions:

- v1alpha1

operations:

- CREATE

- UPDATE

resources:

- mapsservers

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-kibana-k8s-elastic-co-v1-kibana

failurePolicy: Ignore

name: elastic-kb-validation-v1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- kibana.k8s.elastic.co

apiVersions:

- v1

operations:

- CREATE

- UPDATE

resources:

- kibanas

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-kibana-k8s-elastic-co-v1beta1-kibana

failurePolicy: Ignore

name: elastic-kb-validation-v1beta1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- kibana.k8s.elastic.co

apiVersions:

- v1beta1

operations:

- CREATE

- UPDATE

resources:

- kibanas

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-autoscaling-k8s-elastic-co-v1alpha1-elasticsearchautoscaler

failurePolicy: Ignore

name: elastic-esa-validation-v1alpha1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- autoscaling.k8s.elastic.co

apiVersions:

- v1alpha1

operations:

- CREATE

- UPDATE

resources:

- elasticsearchautoscalers

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-scp-k8s-elastic-co-v1alpha1-stackconfigpolicies

failurePolicy: Ignore

name: elastic-scp-validation-v1alpha1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- stackconfigpolicy.k8s.elastic.co

apiVersions:

- v1alpha1

operations:

- CREATE

- UPDATE

resources:

- stackconfigpolicies

clientConfig:

service:

name: elastic-webhook-server

namespace: app-stats-operator-es-prod

path: /validate-logstash-k8s-elastic-co-v1alpha1-logstash

failurePolicy: Ignore

name: elastic-logstash-validation-v1alpha1.k8s.elastic.co

matchPolicy: Exact

admissionReviewVersions: [v1, v1beta1]

sideEffects: None

rules:

- apiGroups:

- logstash.k8s.elastic.co

apiVersions:

- v1alpha1

operations:

- CREATE

- UPDATE

resources:

- logstashes


Elasticsearch

Created in namespace app-stats-es-prod

Conf:

apiVersion: elasticsearch.k8s.elastic.co/v1

kind: Elasticsearch

metadata:

name: es-prod

namespace: app-stats-es-prod

spec:

version: 9.2.4

image: inasoft-harbor.jetpulp.hosting/hub/elastic/elasticsearch:9.2.4

nodeSets:

- name: default

count: 1

config:

node.store.allow_mmap: false

xpack.security.authc.token.enabled: true

# xpack.license.self_generated.type: trial

xpack.security.authc.realms.oidc.oidc1.order: 2

xpack.security.authc.realms.oidc.oidc1.rp.client_id: kibana-stats

xpack.security.authc.realms.oidc.oidc1.rp.response_type: code

xpack.security.authc.realms.oidc.oidc1.rp.redirect_uri: https://kibana-stats.inasoft.net/api/security/oidc/callback

xpack.security.authc.realms.oidc.oidc1.op.issuer: https://keycloak-pp.inasoft.net/realms/Inasoft

xpack.security.authc.realms.oidc.oidc1.op.authorization_endpoint: https://keycloak-pp.inasoft.net/realms/inasoft/protocol/openid-connect/auth

xpack.security.authc.realms.oidc.oidc1.op.token_endpoint: https://keycloak-pp.inasoft.net/realms/inasoft/protocol/openid-connect/token

xpack.security.authc.realms.oidc.oidc1.op.jwkset_path: https://keycloak-pp.inasoft.net/realms/Inasoft/protocol/openid-connect/certs

xpack.security.authc.realms.oidc.oidc1.op.userinfo_endpoint: https://keycloak-pp.inasoft.net/realms/Inasoft/protocol/openid-connect/userinfo

xpack.security.authc.realms.oidc.oidc1.op.endsession_endpoint: https://keycloak-pp.inasoft.net/realms/Inasoft/protocol/openid-connect/logout

xpack.security.authc.realms.oidc.oidc1.rp.post_logout_redirect_uri: https://kibana-stats.inasoft.net/security/logged_out

xpack.security.authc.realms.oidc.oidc1.claims.principal: sub

xpack.security.authc.realms.oidc.oidc1.claims.groups: https://keycloak-pp.inasoft.net/realms/Inasoft/claims/groups

podTemplate:

spec:

imagePullSecrets:

- name: hub-proxy-pull-secret

volumeClaimTemplates:

- metadata:

name: elasticsearch-data

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 5Gi

storageClassName: "netapp-ssd-nfs-sc"

secureSettings:

- secretName: es-secret

http:

service:

spec:

type: ClusterIP

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

annotations:

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

name: es-prod

namespace: app-stats-es-prod

spec:

rules:

- host: es.inasoft.net

http:

paths:

- backend:

service:

name: es-prod-es-http

port:

number: 9200

path: /

pathType: Prefix

tls:

- hosts:

- es.inasoft.net

secretName: wildcard-inasoft-net-tls

status:

loadBalancer:

ingress:

- ip: 10.200.2.49

- ip: 10.200.2.50


elasticsearch.yml contains:

xpack:
  license:
    upload:
      types:
        - trial
        - enterprise

Elasticsearch log extract of the first startup

{"@timestamp":"2026-02-03T10:56:20.664Z","log.level":"DEBUG","message":"state recovered: previous license [null]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:20.665Z","log.level":"DEBUG","message":"state recovered: current license [LicensesMetadata{license={"uid":"6053a19d-e5d4-42ff-8377-406efb6c7a11","type":"basic","issue_date_in_millis":1770116180536,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAODnfp8qET3Z86NqzkbgGoUyloD6BZfXARMIXJ/WlbBJ0ZeUInNFnchD7afN4A+jjPWD5j95mIfUcdlTJHVdk9NpPTFMKL5wC7BXn4CTS6kNo0USZyoItHUPNPNMwVpizHsGj8ctR8OF5A499zxnZ13oe15eFhAb9H2A5MS3hX0HG9uTxyEidFCyu4gWV4qrZJCpCPJibTS35ix4TJbTTfYGlgLYjh0hYpTBh2rh7xfXRa9uXBDfPPOvckJnO5J6vnbtt4h3/Za6edIVyjPdN6c8QHEkjhr52UFZOmbBnjluoQ==","start_date_in_millis":-1}, trialVersion=null}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:20.690Z","log.level": "INFO","message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/file1,native/native1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:20.691Z","log.level": "WARN","message":"The [oidc.oidc1] realm has been automatically disabled due to a change in license [active basic license]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}

…

{"@timestamp":"2026-02-03T10:56:20.973Z","log.level":"DEBUG","message":"executing cluster state update for [started trial license[org.elasticsearch.license.StartTrialClusterTask@359f0acf, org.elasticsearch.license.StartTrialClusterTask@30ad71c6]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.000Z","log.level":"DEBUG","message":"took [27ms] to compute cluster state update for [started trial license[org.elasticsearch.license.StartTrialClusterTask@359f0acf, org.elasticsearch.license.StartTrialClusterTask@30ad71c6]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.000Z","log.level":"DEBUG","message":"cluster state updated, version [22], source [started trial license[org.elasticsearch.license.StartTrialClusterTask@359f0acf, org.elasticsearch.license.StartTrialClusterTask@30ad71c6]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.364Z","log.level":"DEBUG","message":"previous [LicensesMetadata{license={"uid":"6053a19d-e5d4-42ff-8377-406efb6c7a11","type":"basic","issue_date_in_millis":1770116180536,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAODnfp8qET3Z86NqzkbgGoUyloD6BZfXARMIXJ/WlbBJ0ZeUInNFnchD7afN4A+jjPWD5j95mIfUcdlTJHVdk9NpPTFMKL5wC7BXn4CTS6kNo0USZyoItHUPNPNMwVpizHsGj8ctR8OF5A499zxnZ13oe15eFhAb9H2A5MS3hX0HG9uTxyEidFCyu4gWV4qrZJCpCPJibTS35ix4TJbTTfYGlgLYjh0hYpTBh2rh7xfXRa9uXBDfPPOvckJnO5J6vnbtt4h3/Za6edIVyjPdN6c8QHEkjhr52UFZOmbBnjluoQ==","start_date_in_millis":-1}, trialVersion=null}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.364Z","log.level":"DEBUG","message":"current [LicensesMetadata{license={"uid":"e913903c-3949-4dfa-9e2f-6df6db963827","type":"trial","issue_date_in_millis":1770116180974,"expiry_date_in_millis":1772708180974,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAQCZAL7B6mHJmud5odQP0mKEr2jDPr4pm3WB7AflcJ47b5UCa9SUlOpH0HoRC9TEa1xx5IlzR6SKg0UkFPPN/+F0PTFMKL5wC7BXn4CTS6kNozAsQv2QSvOA5Ow1pr7BJTkucvt7DORHXeNfQeCDWi7FCeJl7M3waO8G35qgY45X1/5rxyRiYgNeTGQo8Avm5Hrr89XxMZTrFhE9H0Ag/nUgK11BjONEN6s8VuWy6ggHWANhRpCq9z59tbBoJE2WSNBOaQ6m/XK3/ykP+f1hidjea0J1V1RfoO9XRDd2DC56NzSFGD++XRhS2njltA8EcMcQI0qDFaYkJmt+y2X+mtEc","start_date_in_millis":-1}, trialVersion=9000000}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.392Z","log.level": "INFO","message":"license mode is [trial], currently licensed security realms are [reserved/reserved,file/file1,native/native1,oidc/oidc1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"license [e913903c-3949-4dfa-9e2f-6df6db963827] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"schedule job [licenseJob] with scheduleTime=[1772103380974] and delay=[1987199581]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"added job [licenseJob]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"schedule job [.license_expiration_job_ccbbced3-6922-4ba6-9452-dd654fcbbfd9] with scheduleTime=[1770548180974] and delay=[431999581]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"added job [.license_expiration_job_ccbbced3-6922-4ba6-9452-dd654fcbbfd9]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"schedule job [.license_expiration_job_95d72a8a-9a63-4f8e-a2e5-6b8eb3e4f0a7] with scheduleTime=[1772708180974] and delay=[2591999581]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"added job [.license_expiration_job_95d72a8a-9a63-4f8e-a2e5-6b8eb3e4f0a7]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level": "INFO","message":"license [e913903c-3949-4dfa-9e2f-6df6db963827] mode [trial] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.396Z","log.level":"DEBUG","message":"started self generated trial license: LicensesMetadata{license={"uid":"6053a19d-e5d4-42ff-8377-406efb6c7a11","type":"basic","issue_date_in_millis":1770116180536,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAODnfp8qET3Z86NqzkbgGoUyloD6BZfXARMIXJ/WlbBJ0ZeUInNFnchD7afN4A+jjPWD5j95mIfUcdlTJHVdk9NpPTFMKL5wC7BXn4CTS6kNo0USZyoItHUPNPNMwVpizHsGj8ctR8OF5A499zxnZ13oe15eFhAb9H2A5MS3hX0HG9uTxyEidFCyu4gWV4qrZJCpCPJibTS35ix4TJbTTfYGlgLYjh0hYpTBh2rh7xfXRa9uXBDfPPOvckJnO5J6vnbtt4h3/Za6edIVyjPdN6c8QHEkjhr52UFZOmbBnjluoQ==","start_date_in_millis":-1}, trialVersion=null}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.397Z","log.level":"DEBUG","message":"started self generated trial license: LicensesMetadata{license={"uid":"e913903c-3949-4dfa-9e2f-6df6db963827","type":"trial","issue_date_in_millis":1770116180974,"expiry_date_in_millis":1772708180974,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAQCZAL7B6mHJmud5odQP0mKEr2jDPr4pm3WB7AflcJ47b5UCa9SUlOpH0HoRC9TEa1xx5IlzR6SKg0UkFPPN/+F0PTFMKL5wC7BXn4CTS6kNozAsQv2QSvOA5Ow1pr7BJTkucvt7DORHXeNfQeCDWi7FCeJl7M3waO8G35qgY45X1/5rxyRiYgNeTGQo8Avm5Hrr89XxMZTrFhE9H0Ag/nUgK11BjONEN6s8VuWy6ggHWANhRpCq9z59tbBoJE2WSNBOaQ6m/XK3/ykP+f1hidjea0J1V1RfoO9XRDd2DC56NzSFGD++XRhS2njltA8EcMcQI0qDFaYkJmt+y2X+mtEc","start_date_in_millis":-1}, trialVersion=9000000}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}

Elasticsearch log extract after pod deletion/recreation

Shows that ES starts using the trial license and then swicthes to a basic one



{"@timestamp":"2026-02-03T10:59:22.382Z","log.level": "INFO","message":"license state changed, now [valid]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][generic][T#4]","log.logger":"org.elasticsearch.xpack.writeloadforecaster.LicensedWriteLoadForecaster","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.782Z","log.level":"DEBUG","message":"state recovered: previous license [null]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.782Z","log.level":"DEBUG","message":"state recovered: current license [LicensesMetadata{license={"uid":"e913903c-3949-4dfa-9e2f-6df6db963827","type":"trial","issue_date_in_millis":1770116180974,"expiry_date_in_millis":1772708180974,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAQCZAL7B6mHJmud5odQP0mKEr2jDPr4pm3WB7AflcJ47b5UCa9SUlOpH0HoRC9TEa1xx5IlzR6SKg0UkFPPN/+F0PTFMKL5wC7BXn4CTS6kNozAsQv2QSvOA5Ow1pr7BJTkucvt7DORHXeNfQeCDWi7FCeJl7M3waO8G35qgY45X1/5rxyRiYgNeTGQo8Avm5Hrr89XxMZTrFhE9H0Ag/nUgK11BjONEN6s8VuWy6ggHWANhRpCq9z59tbBoJE2WSNBOaQ6m/XK3/ykP+f1hidjea0J1V1RfoO9XRDd2DC56NzSFGD++XRhS2njltA8EcMcQI0qDFaYkJmt+y2X+mtEc","start_date_in_millis":-1}, trialVersion=9000000}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.831Z","log.level": "INFO","message":"license mode is [trial], currently licensed security realms are [reserved/reserved,file/file1,native/native1,oidc/oidc1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.832Z","log.level":"DEBUG","message":"license [e913903c-3949-4dfa-9e2f-6df6db963827] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.832Z","log.level":"DEBUG","message":"schedule job [licenseJob] with scheduleTime=[1772103380974] and delay=[1987018142]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.833Z","log.level":"DEBUG","message":"added job [licenseJob]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level":"DEBUG","message":"schedule job [.license_expiration_job_09e52f6a-fb37-4642-b359-82f5bca9724b] with scheduleTime=[1770548180974] and delay=[431818140]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level":"DEBUG","message":"added job [.license_expiration_job_09e52f6a-fb37-4642-b359-82f5bca9724b]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level":"DEBUG","message":"schedule job [.license_expiration_job_c21da2cb-c604-4119-8ac9-ae3987d2c66f] with scheduleTime=[1772708180974] and delay=[2591818140]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level":"DEBUG","message":"added job [.license_expiration_job_c21da2cb-c604-4119-8ac9-ae3987d2c66f]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level": "INFO","message":"license [e913903c-3949-4dfa-9e2f-6df6db963827] mode [trial] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.742Z","log.level":"DEBUG","message":"executing cluster state update for [start basic license[org.elasticsearch.license.StartBasicClusterTask@494665c9, org.elasticsearch.license.StartBasicClusterTask@1709c591]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.762Z","log.level":"DEBUG","message":"took [21ms] to compute cluster state update for [start basic license[org.elasticsearch.license.StartBasicClusterTask@494665c9, org.elasticsearch.license.StartBasicClusterTask@1709c591]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.762Z","log.level":"DEBUG","message":"cluster state updated, version [1368], source [start basic license[org.elasticsearch.license.StartBasicClusterTask@494665c9, org.elasticsearch.license.StartBasicClusterTask@1709c591]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.885Z","log.level":"DEBUG","message":"previous [LicensesMetadata{license={"uid":"e913903c-3949-4dfa-9e2f-6df6db963827","type":"trial","issue_date_in_millis":1770116180974,"expiry_date_in_millis":1772708180974,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAQCZAL7B6mHJmud5odQP0mKEr2jDPr4pm3WB7AflcJ47b5UCa9SUlOpH0HoRC9TEa1xx5IlzR6SKg0UkFPPN/+F0PTFMKL5wC7BXn4CTS6kNozAsQv2QSvOA5Ow1pr7BJTkucvt7DORHXeNfQeCDWi7FCeJl7M3waO8G35qgY45X1/5rxyRiYgNeTGQo8Avm5Hrr89XxMZTrFhE9H0Ag/nUgK11BjONEN6s8VuWy6ggHWANhRpCq9z59tbBoJE2WSNBOaQ6m/XK3/ykP+f1hidjea0J1V1RfoO9XRDd2DC56NzSFGD++XRhS2njltA8EcMcQI0qDFaYkJmt+y2X+mtEc","start_date_in_millis":-1}, trialVersion=9000000}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.885Z","log.level":"DEBUG","message":"current [LicensesMetadata{license={"uid":"cb78458d-105a-4f18-a678-3d94845d6a99","type":"basic","issue_date_in_millis":1770116363742,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAOB89D4HO7AE5n2P2n8banuUocUe6NNLdzVHeNR1C6g/AIZ3txSg6TERzVNzYWgZn5qD5j95mIfUcdlTJHVdk9NpPTFMKL5wC7BXn4CTS6kNo1XN2bdL2mRofUt6x90l2v9XGO1SbFSQT3odk3R8DnHAe15eFhAb9H2A5MS3hX0HG9uTxyEidFCyu4gWV4qrZJCpCPJibTS35ix4TJbTTfYGlgLYjh0hYpTBh2rh7xfXRa9uXBDfPPOvckJnO5J6vnbtt4h3/Za6edIVyjPdN6c8QHEkjhr52UFZOmbBnjluoQ==","start_date_in_millis":-1}, trialVersion=9000000}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.910Z","log.level": "INFO","message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/file1,native/native1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.910Z","log.level": "WARN","message":"The [oidc.oidc1] realm has been automatically disabled due to a change in license [active basic license]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.911Z","log.level":"DEBUG","message":"license [cb78458d-105a-4f18-a678-3d94845d6a99] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}