Hello everyone. Yesterday, I successfully deployed and configured Elasticsearch on Kubernetes. We need to use cross-cluster replication, so a trial period was activated. Everything was working fine for about 20 hours, then one node started randomly disconnecting from the master. It was suspected that it lacked resources, so I doubled them and updated the cluster. After that, the trial license became unavailable, and cross-cluster replication stopped working. This has happened twice now. We plan to purchase a platinum license, but I'm unsure about it because even the trial cannot last its full 30 days. It's all very strange; if we can't even try for a few days without having to recreate the clusters each time.
Elasticsearch version: 8.11.3
Logs when the node started randomly disconnecting:
{"@timestamp":"2024-03-19T08:55:31.341Z", "log.level": "INFO", "message":"node-join[{sfo3-es-sfo3-1}{RMyL7_UaTdaH54YrVrKclA}{fdZpfx5LRCegwNpxDstusg}{sfo3-es-sfo3-1}{10.244.6.107}{10.244.6.107:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003} joining, removed [2.8s/2803ms] ago with reason [disconnected], [25] total removals], term: 1, version: 1076, delta: added {{sfo3-es-sfo3-1}{RMyL7_UaTdaH54YrVrKclA}{fdZpfx5LRCegwNpxDstusg}{sfo3-es-sfo3-1}{10.244.6.107}{10.244.6.107:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[sfo3-es-sfo3-0][masterService#updateTask][T#348]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"w4SNQMnsSg2eWYXm51JQPQ","elasticsearch.node.id":"2QvL3D6eQnST1mQhbZEdpg","elasticsearch.node.name":"sfo3-es-sfo3-0","elasticsearch.cluster.name":"sfo3"}
Elasticsearch manifests:
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: sfo3
namespace: elasticsearch
spec:
version: 8.11.3
http:
tls:
selfSignedCertificate:
disabled: true
transport:
tls:
certificateAuthorities:
configMapName: remote-certs
service:
spec:
type: LoadBalancer
nodeSets:
- name: sfo3
count: 2
config:
node.store.allow_mmap: False
podTemplate:
spec:
containers:
- name: elasticsearch
resources:
requests:
memory: 2Gi
cpu: 2
limits:
memory: 2Gi
Cluster real age:
kubectl get sts -n elasticsearch
NAME READY AGE
sfo3-es-sfo3 2/2 26h
API response to create follower index:
Error creating follower index: [403 Forbidden] {
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "current license is non-compliant for [ccr]",
"license.expired.feature" : "ccr"
}
],
"type" : "security_exception",
"reason" : "current license is non-compliant for [ccr]",
"license.expired.feature" : "ccr"
},
"status" : 403
}
Checking the license on the main cluster:
curl -u elastic:$PW -XGET http://$REGION-es-http:9200/_license
{
"license" : {
"status" : "active",
"uid" : "918a3d9e-4e73-48b5-b970-0a38b2619db6",
"type" : "basic",
"issue_date" : "2024-03-19T10:15:12.734Z",
"issue_date_in_millis" : 1710843312734,
"max_nodes" : 1000,
"max_resource_units" : null,
"issued_to" : "sfo3",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}
curl -u elastic:$PW -X POST "http://$REGION-es-http:9200/_license/start_trial?acknowledge=true&pretty"
{
"acknowledged" : true,
"trial_was_started" : false,
"error_message" : "Operation failed: Trial was already activated."
}
Checking the license on the replica cluster:
curl -u elastic:$PW -XGET http://$REGION-es-http:9200/_license
{
"license" : {
"status" : "active",
"uid" : "423276a7-9e26-400e-8f57-cf1f3bc94a40",
"type" : "trial",
"issue_date" : "2024-03-18T07:51:38.770Z",
"issue_date_in_millis" : 1710748298770,
"expiry_date" : "2024-04-17T07:51:38.770Z",
"expiry_date_in_millis" : 1713340298770,
"max_nodes" : 1000,
"max_resource_units" : null,
"issued_to" : "fra1",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}
Let me know if I can provide any more important information, because it's just a development cluster without any clients. I hope for mutual assistance in resolving this issue. Thank you.