I have deployed a ES stack on Kubernetes with the operator (ECK). Just ES + Kibana, and it is Ok.
I’m trying to test some enterprise functions, and activate a trial licence, but elasticsearch reverts to a basic license after each configuration change or pod delete/create
What I did in summary
- Create a yaml file for the operator to activate the license
- kubectl apply to activate it
- Create the Elasticsearch service (Pod) named es-prod
What I see
- The operator activates the license.
- I see a ConfigMap elastic-licensing with a valid license
- Elasticsearch starts, and has a valid trial license. Checked on the created secret named es-prod-es-license, and with an API call on /_license
- I see a lot of creations / changes on the secret named es-prod-es-license (the uid changes)
- If I change ES conf or delete the Pod (which is then recreated), ES starts with the trial license, and replaces it by the basic license
Some technical infos
Operator
Namespace: app-stats-operator-es-prod
Conf:
Source: eck-operator/templates/service-account.yaml
apiVersion: v1
kind: ServiceAccount
automountServiceAccountToken: true
metadata:
name: elastic-stats-prod-operator
namespace: app-stats-operator-es-prod
labels:
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
Source: eck-operator/templates/webhook.yaml
apiVersion: v1
kind: Secret
metadata:
name: elastic-webhook-server-cert
namespace: app-stats-operator-es-prod
labels:
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
Source: eck-operator/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: elastic-stats-prod-operator
namespace: app-stats-operator-es-prod
labels:
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
data:
eck.yaml: |-
log-verbosity: 0
metrics-port: 0
metrics-secure: false
container-registry: docker.elastic.co
max-concurrent-reconciles: 3
ca-cert-validity: 8760h
ca-cert-rotate-before: 24h
cert-validity: 8760h
cert-rotate-before: 24h
disable-config-watch: false
exposed-node-labels: [topology.kubernetes.io/.,failure-domain.beta.kubernetes.io/.]
set-default-security-context: auto-detect
kube-client-timeout: 60s
elasticsearch-client-timeout: 180s
disable-telemetry: false
distribution-channel: all-in-one
validate-storage-class: true
enable-webhook: true
webhook-name: elastic-webhook-prod.k8s.elastic.co
webhook-port: 9443
operator-namespace: app-stats-operator-es-prod
namespaces: app-stats-es-prod,app-stats-kibana-prod,app-stats-logstash-prod,app-stats-operator-es-prod
enable-leader-election: true
elasticsearch-observation-interval: 10s
ubi-only: false
Source: eck-operator/templates/cluster-roles.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: elastic-stats-prod-operator
labels:
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
rules:
apiGroups:
- "authorization.k8s.io"
resources:
- subjectaccessreviews
verbs:
- create
apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
apiGroups:
- coordination.k8s.io
resources:
- leases
resourceNames:
- elastic-operator-leader
verbs:
- get
- watch
- update
apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- list
- watch
apiGroups:
- ""
resources:
- pods
- events
- persistentvolumeclaims
- secrets
- services
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
apiGroups:
- apps
resources:
- deployments
- statefulsets
- daemonsets
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
apiGroups:
- elasticsearch.k8s.elastic.co
resources:
- elasticsearches
- elasticsearches/status
- elasticsearches/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- autoscaling.k8s.elastic.co
resources:
- elasticsearchautoscalers
- elasticsearchautoscalers/status
- elasticsearchautoscalers/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- kibana.k8s.elastic.co
resources:
- kibanas
- kibanas/status
- kibanas/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- apm.k8s.elastic.co
resources:
- apmservers
- apmservers/status
- apmservers/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- enterprisesearch.k8s.elastic.co
resources:
- enterprisesearches
- enterprisesearches/status
- enterprisesearches/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- beat.k8s.elastic.co
resources:
- beats
- beats/status
- beats/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- agent.k8s.elastic.co
resources:
- agents
- agents/status
- agents/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- maps.k8s.elastic.co
resources:
- elasticmapsservers
- elasticmapsservers/status
- elasticmapsservers/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- stackconfigpolicy.k8s.elastic.co
resources:
- stackconfigpolicies
- stackconfigpolicies/status
- stackconfigpolicies/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- logstash.k8s.elastic.co
resources:
- logstashes
- logstashes/status
- logstashes/finalizers # needed for ownerReferences with blockOwnerDeletion on OCP
verbs:
- get
- list
- watch
- create
- update
- patch
apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- list
- watch
apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
Source: eck-operator/templates/cluster-roles.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: "elastic-stats-prod-operator-view"
labels:
rbac.authorization.k8s.io/aggregate-to-view: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-admin: "true"
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
rules:
apiGroups: ["elasticsearch.k8s.elastic.co"]
resources: ["elasticsearches"]
verbs: ["get", "list", "watch"]
apiGroups: ["autoscaling.k8s.elastic.co"]
resources: ["elasticsearchautoscalers"]
verbs: ["get", "list", "watch"]
apiGroups: ["apm.k8s.elastic.co"]
resources: ["apmservers"]
verbs: ["get", "list", "watch"]
apiGroups: ["kibana.k8s.elastic.co"]
resources: ["kibanas"]
verbs: ["get", "list", "watch"]
apiGroups: ["enterprisesearch.k8s.elastic.co"]
resources: ["enterprisesearches"]
verbs: ["get", "list", "watch"]
apiGroups: ["beat.k8s.elastic.co"]
resources: ["beats"]
verbs: ["get", "list", "watch"]
apiGroups: ["agent.k8s.elastic.co"]
resources: ["agents"]
verbs: ["get", "list", "watch"]
apiGroups: ["maps.k8s.elastic.co"]
resources: ["elasticmapsservers"]
verbs: ["get", "list", "watch"]
apiGroups: ["stackconfigpolicy.k8s.elastic.co"]
resources: ["stackconfigpolicies"]
verbs: ["get", "list", "watch"]
apiGroups: ["logstash.k8s.elastic.co"]
resources: ["logstashes"]
verbs: ["get", "list", "watch"]
Source: eck-operator/templates/cluster-roles.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: "elastic-stats-prod-operator-edit"
labels:
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-admin: "true"
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
rules:
apiGroups: ["elasticsearch.k8s.elastic.co"]
resources: ["elasticsearches"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
apiGroups: ["autoscaling.k8s.elastic.co"]
resources: ["elasticsearchautoscalers"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
apiGroups: ["apm.k8s.elastic.co"]
resources: ["apmservers"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
apiGroups: ["kibana.k8s.elastic.co"]
resources: ["kibanas"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
apiGroups: ["enterprisesearch.k8s.elastic.co"]
resources: ["enterprisesearches"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
apiGroups: ["beat.k8s.elastic.co"]
resources: ["beats"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
apiGroups: ["agent.k8s.elastic.co"]
resources: ["agents"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
apiGroups: ["maps.k8s.elastic.co"]
resources: ["elasticmapsservers"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
apiGroups: ["stackconfigpolicy.k8s.elastic.co"]
resources: ["stackconfigpolicies"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
apiGroups: ["logstash.k8s.elastic.co"]
resources: ["logstashes"]
verbs: ["create", "delete", "deletecollection", "patch", "update"]
Source: eck-operator/templates/role-bindings.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: elastic-stats-prod-operator
labels:
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: elastic-stats-prod-operator
subjects:
kind: ServiceAccount
name: elastic-stats-prod-operator
namespace: app-stats-operator-es-prod
Source: eck-operator/templates/webhook.yaml
apiVersion: v1
kind: Service
metadata:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
labels:
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
spec:
ports:
- name: https
port: 443
targetPort: 9443
selector:
control-plane: elastic-stats-prod-operator
Source: eck-operator/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elastic-stats-prod-operator
namespace: app-stats-operator-es-prod
labels:
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
spec:
selector:
matchLabels:
control-plane: elastic-stats-prod-operator
serviceName: elastic-stats-prod-operator
replicas: 1
template:
metadata:
annotations:
# Rename the fields "error" to "error.message" and "source" to "event.source"
# This is to avoid a conflict with the ECS "error" and "source" documents.
"co.elastic.logs/raw": "[{"type":"container","json.keys_under_root":true,"paths":["/var/log/containers/*${data.kubernetes.container.id}.log"],"processors":[{"convert":{"mode":"rename","ignore_missing":true,"fields":[{"from":"error","to":"_error"}]}},{"convert":{"mode":"rename","ignore_missing":true,"fields":[{"from":"_error","to":"error.message"}]}},{"convert":{"mode":"rename","ignore_missing":true,"fields":[{"from":"source","to":"_source"}]}},{"convert":{"mode":"rename","ignore_missing":true,"fields":[{"from":"_source","to":"event.source"}]}}]}]"
"checksum/config": 3fdaff4a979089797cb469806c19162f44297428bf811896931864c528196029
labels:
control-plane: elastic-stats-prod-operator
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: elastic-stats-prod-operator
automountServiceAccountToken: true
securityContext:
runAsNonRoot: true
containers:
- image: "inasoft-harbor.jetpulp.hosting/hub/elastic/eck-operator:3.2.0"
imagePullPolicy: IfNotPresent
name: manager
args:
- "manager"
- "--config=/conf/eck.yaml"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
env:
- name: OPERATOR_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: WEBHOOK_SECRET
value: elastic-webhook-server-cert
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 100m
memory: 150Mi
ports:
- containerPort: 9443
name: https-webhook
protocol: TCP
volumeMounts:
- mountPath: "/conf"
name: conf
readOnly: true
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
volumes:
- name: conf
configMap:
name: elastic-stats-prod-operator
- name: cert
secret:
defaultMode: 420
secretName: elastic-webhook-server-cert
imagePullSecrets:
- name: hub-proxy-pull-secret
Source: eck-operator/templates/webhook.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: elastic-webhook-prod.k8s.elastic.co
labels:
control-plane: elastic-stats-prod-operator
app.kubernetes.io/version: "3.2.0"
webhooks:
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-agent-k8s-elastic-co-v1alpha1-agent
failurePolicy: Ignore
name: elastic-agent-validation-v1alpha1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- agent.k8s.elastic.co
apiVersions:
- v1alpha1
operations:
- CREATE
- UPDATE
resources:
- agents
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-apm-k8s-elastic-co-v1-apmserver
failurePolicy: Ignore
name: elastic-apm-validation-v1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- apm.k8s.elastic.co
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- apmservers
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-apm-k8s-elastic-co-v1beta1-apmserver
failurePolicy: Ignore
name: elastic-apm-validation-v1beta1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- apm.k8s.elastic.co
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- apmservers
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-beat-k8s-elastic-co-v1beta1-beat
failurePolicy: Ignore
name: elastic-beat-validation-v1beta1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- beat.k8s.elastic.co
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- beats
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-enterprisesearch-k8s-elastic-co-v1-enterprisesearch
failurePolicy: Ignore
name: elastic-ent-validation-v1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- enterprisesearch.k8s.elastic.co
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- enterprisesearches
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-enterprisesearch-k8s-elastic-co-v1beta1-enterprisesearch
failurePolicy: Ignore
name: elastic-ent-validation-v1beta1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- enterprisesearch.k8s.elastic.co
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- enterprisesearches
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-elasticsearch-k8s-elastic-co-v1-elasticsearch
failurePolicy: Ignore
name: elastic-es-validation-v1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- elasticsearch.k8s.elastic.co
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- elasticsearches
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-elasticsearch-k8s-elastic-co-v1beta1-elasticsearch
failurePolicy: Ignore
name: elastic-es-validation-v1beta1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- elasticsearch.k8s.elastic.co
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- elasticsearches
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-ems-k8s-elastic-co-v1alpha1-mapsservers
failurePolicy: Ignore
name: elastic-ems-validation-v1alpha1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- maps.k8s.elastic.co
apiVersions:
- v1alpha1
operations:
- CREATE
- UPDATE
resources:
- mapsservers
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-kibana-k8s-elastic-co-v1-kibana
failurePolicy: Ignore
name: elastic-kb-validation-v1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- kibana.k8s.elastic.co
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- kibanas
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-kibana-k8s-elastic-co-v1beta1-kibana
failurePolicy: Ignore
name: elastic-kb-validation-v1beta1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- kibana.k8s.elastic.co
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- kibanas
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-autoscaling-k8s-elastic-co-v1alpha1-elasticsearchautoscaler
failurePolicy: Ignore
name: elastic-esa-validation-v1alpha1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- autoscaling.k8s.elastic.co
apiVersions:
- v1alpha1
operations:
- CREATE
- UPDATE
resources:
- elasticsearchautoscalers
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-scp-k8s-elastic-co-v1alpha1-stackconfigpolicies
failurePolicy: Ignore
name: elastic-scp-validation-v1alpha1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- stackconfigpolicy.k8s.elastic.co
apiVersions:
- v1alpha1
operations:
- CREATE
- UPDATE
resources:
- stackconfigpolicies
clientConfig:
service:
name: elastic-webhook-server
namespace: app-stats-operator-es-prod
path: /validate-logstash-k8s-elastic-co-v1alpha1-logstash
failurePolicy: Ignore
name: elastic-logstash-validation-v1alpha1.k8s.elastic.co
matchPolicy: Exact
admissionReviewVersions: [v1, v1beta1]
sideEffects: None
rules:
- apiGroups:
- logstash.k8s.elastic.co
apiVersions:
- v1alpha1
operations:
- CREATE
- UPDATE
resources:
- logstashes
Elasticsearch
Created in namespace app-stats-es-prod
Conf:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: es-prod
namespace: app-stats-es-prod
spec:
version: 9.2.4
image: inasoft-harbor.jetpulp.hosting/hub/elastic/elasticsearch:9.2.4
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: false
xpack.security.authc.token.enabled: true
# xpack.license.self_generated.type: trial
xpack.security.authc.realms.oidc.oidc1.order: 2
xpack.security.authc.realms.oidc.oidc1.rp.client_id: kibana-stats
xpack.security.authc.realms.oidc.oidc1.rp.response_type: code
xpack.security.authc.realms.oidc.oidc1.rp.redirect_uri: https://kibana-stats.inasoft.net/api/security/oidc/callback
xpack.security.authc.realms.oidc.oidc1.op.issuer: https://keycloak-pp.inasoft.net/realms/Inasoft
xpack.security.authc.realms.oidc.oidc1.op.authorization_endpoint: https://keycloak-pp.inasoft.net/realms/inasoft/protocol/openid-connect/auth
xpack.security.authc.realms.oidc.oidc1.op.token_endpoint: https://keycloak-pp.inasoft.net/realms/inasoft/protocol/openid-connect/token
xpack.security.authc.realms.oidc.oidc1.op.jwkset_path: https://keycloak-pp.inasoft.net/realms/Inasoft/protocol/openid-connect/certs
xpack.security.authc.realms.oidc.oidc1.op.userinfo_endpoint: https://keycloak-pp.inasoft.net/realms/Inasoft/protocol/openid-connect/userinfo
xpack.security.authc.realms.oidc.oidc1.op.endsession_endpoint: https://keycloak-pp.inasoft.net/realms/Inasoft/protocol/openid-connect/logout
xpack.security.authc.realms.oidc.oidc1.rp.post_logout_redirect_uri: https://kibana-stats.inasoft.net/security/logged_out
xpack.security.authc.realms.oidc.oidc1.claims.principal: sub
xpack.security.authc.realms.oidc.oidc1.claims.groups: https://keycloak-pp.inasoft.net/realms/Inasoft/claims/groups
podTemplate:
spec:
imagePullSecrets:
- name: hub-proxy-pull-secret
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: "netapp-ssd-nfs-sc"
secureSettings:
- secretName: es-secret
http:
service:
spec:
type: ClusterIP
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
name: es-prod
namespace: app-stats-es-prod
spec:
rules:
- host: es.inasoft.net
http:
paths:
- backend:
service:
name: es-prod-es-http
port:
number: 9200
path: /
pathType: Prefix
tls:
- hosts:
- es.inasoft.net
secretName: wildcard-inasoft-net-tls
status:
loadBalancer:
ingress:
- ip: 10.200.2.49
- ip: 10.200.2.50
elasticsearch.yml contains:
xpack:
license:
upload:
types:
- trial
- enterprise
Elasticsearch log extract of the first startup
{"@timestamp":"2026-02-03T10:56:20.664Z","log.level":"DEBUG","message":"state recovered: previous license [null]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:20.665Z","log.level":"DEBUG","message":"state recovered: current license [LicensesMetadata{license={"uid":"6053a19d-e5d4-42ff-8377-406efb6c7a11","type":"basic","issue_date_in_millis":1770116180536,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAODnfp8qET3Z86NqzkbgGoUyloD6BZfXARMIXJ/WlbBJ0ZeUInNFnchD7afN4A+jjPWD5j95mIfUcdlTJHVdk9NpPTFMKL5wC7BXn4CTS6kNo0USZyoItHUPNPNMwVpizHsGj8ctR8OF5A499zxnZ13oe15eFhAb9H2A5MS3hX0HG9uTxyEidFCyu4gWV4qrZJCpCPJibTS35ix4TJbTTfYGlgLYjh0hYpTBh2rh7xfXRa9uXBDfPPOvckJnO5J6vnbtt4h3/Za6edIVyjPdN6c8QHEkjhr52UFZOmbBnjluoQ==","start_date_in_millis":-1}, trialVersion=null}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:20.690Z","log.level": "INFO","message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/file1,native/native1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:20.691Z","log.level": "WARN","message":"The [oidc.oidc1] realm has been automatically disabled due to a change in license [active basic license]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
…
{"@timestamp":"2026-02-03T10:56:20.973Z","log.level":"DEBUG","message":"executing cluster state update for [started trial license[org.elasticsearch.license.StartTrialClusterTask@359f0acf, org.elasticsearch.license.StartTrialClusterTask@30ad71c6]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.000Z","log.level":"DEBUG","message":"took [27ms] to compute cluster state update for [started trial license[org.elasticsearch.license.StartTrialClusterTask@359f0acf, org.elasticsearch.license.StartTrialClusterTask@30ad71c6]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.000Z","log.level":"DEBUG","message":"cluster state updated, version [22], source [started trial license[org.elasticsearch.license.StartTrialClusterTask@359f0acf, org.elasticsearch.license.StartTrialClusterTask@30ad71c6]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.364Z","log.level":"DEBUG","message":"previous [LicensesMetadata{license={"uid":"6053a19d-e5d4-42ff-8377-406efb6c7a11","type":"basic","issue_date_in_millis":1770116180536,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAODnfp8qET3Z86NqzkbgGoUyloD6BZfXARMIXJ/WlbBJ0ZeUInNFnchD7afN4A+jjPWD5j95mIfUcdlTJHVdk9NpPTFMKL5wC7BXn4CTS6kNo0USZyoItHUPNPNMwVpizHsGj8ctR8OF5A499zxnZ13oe15eFhAb9H2A5MS3hX0HG9uTxyEidFCyu4gWV4qrZJCpCPJibTS35ix4TJbTTfYGlgLYjh0hYpTBh2rh7xfXRa9uXBDfPPOvckJnO5J6vnbtt4h3/Za6edIVyjPdN6c8QHEkjhr52UFZOmbBnjluoQ==","start_date_in_millis":-1}, trialVersion=null}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.364Z","log.level":"DEBUG","message":"current [LicensesMetadata{license={"uid":"e913903c-3949-4dfa-9e2f-6df6db963827","type":"trial","issue_date_in_millis":1770116180974,"expiry_date_in_millis":1772708180974,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAQCZAL7B6mHJmud5odQP0mKEr2jDPr4pm3WB7AflcJ47b5UCa9SUlOpH0HoRC9TEa1xx5IlzR6SKg0UkFPPN/+F0PTFMKL5wC7BXn4CTS6kNozAsQv2QSvOA5Ow1pr7BJTkucvt7DORHXeNfQeCDWi7FCeJl7M3waO8G35qgY45X1/5rxyRiYgNeTGQo8Avm5Hrr89XxMZTrFhE9H0Ag/nUgK11BjONEN6s8VuWy6ggHWANhRpCq9z59tbBoJE2WSNBOaQ6m/XK3/ykP+f1hidjea0J1V1RfoO9XRDd2DC56NzSFGD++XRhS2njltA8EcMcQI0qDFaYkJmt+y2X+mtEc","start_date_in_millis":-1}, trialVersion=9000000}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.392Z","log.level": "INFO","message":"license mode is [trial], currently licensed security realms are [reserved/reserved,file/file1,native/native1,oidc/oidc1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"license [e913903c-3949-4dfa-9e2f-6df6db963827] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"schedule job [licenseJob] with scheduleTime=[1772103380974] and delay=[1987199581]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"added job [licenseJob]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"schedule job [.license_expiration_job_ccbbced3-6922-4ba6-9452-dd654fcbbfd9] with scheduleTime=[1770548180974] and delay=[431999581]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"added job [.license_expiration_job_ccbbced3-6922-4ba6-9452-dd654fcbbfd9]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"schedule job [.license_expiration_job_95d72a8a-9a63-4f8e-a2e5-6b8eb3e4f0a7] with scheduleTime=[1772708180974] and delay=[2591999581]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level":"DEBUG","message":"added job [.license_expiration_job_95d72a8a-9a63-4f8e-a2e5-6b8eb3e4f0a7]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.393Z","log.level": "INFO","message":"license [e913903c-3949-4dfa-9e2f-6df6db963827] mode [trial] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.396Z","log.level":"DEBUG","message":"started self generated trial license: LicensesMetadata{license={"uid":"6053a19d-e5d4-42ff-8377-406efb6c7a11","type":"basic","issue_date_in_millis":1770116180536,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAODnfp8qET3Z86NqzkbgGoUyloD6BZfXARMIXJ/WlbBJ0ZeUInNFnchD7afN4A+jjPWD5j95mIfUcdlTJHVdk9NpPTFMKL5wC7BXn4CTS6kNo0USZyoItHUPNPNMwVpizHsGj8ctR8OF5A499zxnZ13oe15eFhAb9H2A5MS3hX0HG9uTxyEidFCyu4gWV4qrZJCpCPJibTS35ix4TJbTTfYGlgLYjh0hYpTBh2rh7xfXRa9uXBDfPPOvckJnO5J6vnbtt4h3/Za6edIVyjPdN6c8QHEkjhr52UFZOmbBnjluoQ==","start_date_in_millis":-1}, trialVersion=null}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
{"@timestamp":"2026-02-03T10:56:21.397Z","log.level":"DEBUG","message":"started self generated trial license: LicensesMetadata{license={"uid":"e913903c-3949-4dfa-9e2f-6df6db963827","type":"trial","issue_date_in_millis":1770116180974,"expiry_date_in_millis":1772708180974,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAQCZAL7B6mHJmud5odQP0mKEr2jDPr4pm3WB7AflcJ47b5UCa9SUlOpH0HoRC9TEa1xx5IlzR6SKg0UkFPPN/+F0PTFMKL5wC7BXn4CTS6kNozAsQv2QSvOA5Ow1pr7BJTkucvt7DORHXeNfQeCDWi7FCeJl7M3waO8G35qgY45X1/5rxyRiYgNeTGQo8Avm5Hrr89XxMZTrFhE9H0Ag/nUgK11BjONEN6s8VuWy6ggHWANhRpCq9z59tbBoJE2WSNBOaQ6m/XK3/ykP+f1hidjea0J1V1RfoO9XRDd2DC56NzSFGD++XRhS2njltA8EcMcQI0qDFaYkJmt+y2X+mtEc","start_date_in_millis":-1}, trialVersion=9000000}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}
Elasticsearch log extract after pod deletion/recreation
Shows that ES starts using the trial license and then swicthes to a basic one
{"@timestamp":"2026-02-03T10:59:22.382Z","log.level": "INFO","message":"license state changed, now [valid]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][generic][T#4]","log.logger":"org.elasticsearch.xpack.writeloadforecaster.LicensedWriteLoadForecaster","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.782Z","log.level":"DEBUG","message":"state recovered: previous license [null]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.782Z","log.level":"DEBUG","message":"state recovered: current license [LicensesMetadata{license={"uid":"e913903c-3949-4dfa-9e2f-6df6db963827","type":"trial","issue_date_in_millis":1770116180974,"expiry_date_in_millis":1772708180974,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAQCZAL7B6mHJmud5odQP0mKEr2jDPr4pm3WB7AflcJ47b5UCa9SUlOpH0HoRC9TEa1xx5IlzR6SKg0UkFPPN/+F0PTFMKL5wC7BXn4CTS6kNozAsQv2QSvOA5Ow1pr7BJTkucvt7DORHXeNfQeCDWi7FCeJl7M3waO8G35qgY45X1/5rxyRiYgNeTGQo8Avm5Hrr89XxMZTrFhE9H0Ag/nUgK11BjONEN6s8VuWy6ggHWANhRpCq9z59tbBoJE2WSNBOaQ6m/XK3/ykP+f1hidjea0J1V1RfoO9XRDd2DC56NzSFGD++XRhS2njltA8EcMcQI0qDFaYkJmt+y2X+mtEc","start_date_in_millis":-1}, trialVersion=9000000}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.831Z","log.level": "INFO","message":"license mode is [trial], currently licensed security realms are [reserved/reserved,file/file1,native/native1,oidc/oidc1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.832Z","log.level":"DEBUG","message":"license [e913903c-3949-4dfa-9e2f-6df6db963827] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.832Z","log.level":"DEBUG","message":"schedule job [licenseJob] with scheduleTime=[1772103380974] and delay=[1987018142]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.833Z","log.level":"DEBUG","message":"added job [licenseJob]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level":"DEBUG","message":"schedule job [.license_expiration_job_09e52f6a-fb37-4642-b359-82f5bca9724b] with scheduleTime=[1770548180974] and delay=[431818140]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level":"DEBUG","message":"added job [.license_expiration_job_09e52f6a-fb37-4642-b359-82f5bca9724b]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level":"DEBUG","message":"schedule job [.license_expiration_job_c21da2cb-c604-4119-8ac9-ae3987d2c66f] with scheduleTime=[1772708180974] and delay=[2591818140]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level":"DEBUG","message":"added job [.license_expiration_job_c21da2cb-c604-4119-8ac9-ae3987d2c66f]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.scheduler.SchedulerEngine","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:22.834Z","log.level": "INFO","message":"license [e913903c-3949-4dfa-9e2f-6df6db963827] mode [trial] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.742Z","log.level":"DEBUG","message":"executing cluster state update for [start basic license[org.elasticsearch.license.StartBasicClusterTask@494665c9, org.elasticsearch.license.StartBasicClusterTask@1709c591]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.762Z","log.level":"DEBUG","message":"took [21ms] to compute cluster state update for [start basic license[org.elasticsearch.license.StartBasicClusterTask@494665c9, org.elasticsearch.license.StartBasicClusterTask@1709c591]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.762Z","log.level":"DEBUG","message":"cluster state updated, version [1368], source [start basic license[org.elasticsearch.license.StartBasicClusterTask@494665c9, org.elasticsearch.license.StartBasicClusterTask@1709c591]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.885Z","log.level":"DEBUG","message":"previous [LicensesMetadata{license={"uid":"e913903c-3949-4dfa-9e2f-6df6db963827","type":"trial","issue_date_in_millis":1770116180974,"expiry_date_in_millis":1772708180974,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAQCZAL7B6mHJmud5odQP0mKEr2jDPr4pm3WB7AflcJ47b5UCa9SUlOpH0HoRC9TEa1xx5IlzR6SKg0UkFPPN/+F0PTFMKL5wC7BXn4CTS6kNozAsQv2QSvOA5Ow1pr7BJTkucvt7DORHXeNfQeCDWi7FCeJl7M3waO8G35qgY45X1/5rxyRiYgNeTGQo8Avm5Hrr89XxMZTrFhE9H0Ag/nUgK11BjONEN6s8VuWy6ggHWANhRpCq9z59tbBoJE2WSNBOaQ6m/XK3/ykP+f1hidjea0J1V1RfoO9XRDd2DC56NzSFGD++XRhS2njltA8EcMcQI0qDFaYkJmt+y2X+mtEc","start_date_in_millis":-1}, trialVersion=9000000}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.885Z","log.level":"DEBUG","message":"current [LicensesMetadata{license={"uid":"cb78458d-105a-4f18-a678-3d94845d6a99","type":"basic","issue_date_in_millis":1770116363742,"max_nodes":1000,"max_resource_units":null,"issued_to":"es-prod","issuer":"elasticsearch","signature":"////+wAAAOB89D4HO7AE5n2P2n8banuUocUe6NNLdzVHeNR1C6g/AIZ3txSg6TERzVNzYWgZn5qD5j95mIfUcdlTJHVdk9NpPTFMKL5wC7BXn4CTS6kNo1XN2bdL2mRofUt6x90l2v9XGO1SbFSQT3odk3R8DnHAe15eFhAb9H2A5MS3hX0HG9uTxyEidFCyu4gWV4qrZJCpCPJibTS35ix4TJbTTfYGlgLYjh0hYpTBh2rh7xfXRa9uXBDfPPOvckJnO5J6vnbtt4h3/Za6edIVyjPdN6c8QHEkjhr52UFZOmbBnjluoQ==","start_date_in_millis":-1}, trialVersion=9000000}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.910Z","log.level": "INFO","message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/file1,native/native1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.910Z","log.level": "WARN","message":"The [oidc.oidc1] realm has been automatically disabled due to a change in license [active basic license]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}{"@timestamp":"2026-02-03T10:59:23.911Z","log.level":"DEBUG","message":"license [cb78458d-105a-4f18-a678-3d94845d6a99] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-prod-es-default-0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.cluster.uuid":"YXe_wkRhQxuRHudEJX0hSw","elasticsearch.node.id":"u4t7Ivf3Sh-6F6Gm2Ebdeg","elasticsearch.node.name":"es-prod-es-default-0","elasticsearch.cluster.name":"${sys:es.logs.cluster_name}"}