How to remove trial license on not running elastic pod

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

When elastic server is down (not the pod) caused by ending trial license how can I remove the trial license from the pod and start elastic?

 elasticsearch {"type": "server", "timestamp": "2020-03-25T09:27:14,081Z", "level": "ERROR", "component": "o.e.x.s.a.f.SecurityActionFilter", "cluster.name": "elastic", "node.name": "elastic-es-default-0", "message": "blocking [cluster:monitor/nodes/stats] operation due t ││
│ elasticsearch {"type": "server", "timestamp": "2020-03-25T09:27:15,848Z", "level": "ERROR", "component": "o.e.x.s.a.f.SecurityActionFilter", "cluster.name": "elastic", "node.name": "elastic-es-default-0", "message": "blocking [cluster:monitor/stats] operation due to expi ││
│ elasticsearch {"type": "server", "timestamp": "2020-03-25T09:27:15,848Z", "level": "ERROR", "component": "o.e.x.m.c.c.ClusterStatsCollector", "cluster.name": "elastic", "node.name": "elastic-es-default-0", "message": "collector [cluster_stats] failed to collect data", "c ││
│ elasticsearch "stacktrace": ["org.elasticsearch.ElasticsearchSecurityException: current license is non-compliant for [security]",
2

Seems the proces is still running but no response on port 9200

[root@elastic-es-default-0 elasticsearch]# ps -ef |grep elastic
elastic+      1      0 34 09:35 ?        00:00:54 /usr/share/elasticsearch/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-5866703941178511907 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Des.cgroups.hierarchy.override=/ -Dio.netty.allocator.type=unpooled -XX:MaxDirectMemorySize=536870912 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch
elastic+     87      1  0 09:35 ?        00:00:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

[root@elastic-es-default-0 elasticsearch]# curl -u elastic -X DELETE "localhost:9200/_license"
Enter host password for user 'elastic':
curl: (52) Empty reply from server
3

You can revert to the basic licence by deleting the trial licence secret.

kubectl delete secret -l 'license.k8s.elastic.co/type=enterprise-trial' -n elastic-system
4

Hi Charith

I thought the license is installed in the Elasticsearch database which is not accessible.
I can't find a secret with the label license.k8s.elastic.co or type enterprise-trial

[lpc-cicd-kube.config ~]$ for d in `k get secret -n l12m-elastic |awk '{print $1}' |grep -v NAME` ; do k describe secret $d -n l12m-elastic| grep license ; done
+ kubectl get secret -n l12m-elastic
+ kubectl describe secret default-token-7sdls -n l12m-elastic
+ kubectl describe secret elastic-es-default-es-config -n l12m-elastic
+ kubectl describe secret elastic-es-elastic-user -n l12m-elastic
+ kubectl describe secret elastic-es-http-ca-internal -n l12m-elastic
+ kubectl describe secret elastic-es-http-certs-internal -n l12m-elastic
+ kubectl describe secret elastic-es-http-certs-public -n l12m-elastic
+ kubectl describe secret elastic-es-internal-users -n l12m-elastic
+ kubectl describe secret elastic-es-transport-ca-internal -n l12m-elastic
+ kubectl describe secret elastic-es-transport-certificates -n l12m-elastic
+ kubectl describe secret elastic-es-transport-certs-public -n l12m-elastic
+ kubectl describe secret elastic-es-xpack-file-realm -n l12m-elastic
+ kubectl describe secret kibana-kb-config -n l12m-elastic
+ kubectl describe secret kibana-kb-es-ca -n l12m-elastic
+ kubectl describe secret kibana-kb-http-ca-internal -n l12m-elastic
+ kubectl describe secret kibana-kb-http-certs-internal -n l12m-elastic
+ kubectl describe secret kibana-kb-http-certs-public -n l12m-elastic
+ kubectl describe secret kibana-kibana-user -n l12m-elastic
+ kubectl describe secret l12m-elastic-kibana-kibana-user -n l12m-elastic
+ kubectl describe secret lpc-cicd-tls -n l12m-elastic
[lpc-cicd-kube.config ~]$ for d in `k get secret -n elastic-system |awk '{print $1}' |grep -v NAME` ; do k describe secret $d -n elastic-system| grep license ; done
+ kubectl get secret -n elastic-system
+ kubectl describe secret default-token-ljgn5 -n elastic-system
+ kubectl describe secret elastic-operator-token-q4dbg -n elastic-system
5

How did you enable the trial in the first place? If you followed the instructions in https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-licensing.html, the secret would have been created in the elastic-system namespace -- which is the default namespace where the operator is installed.

6

Through Kibana

7

In that case, you should be able to revert to basic using the Kibana licence management UI as well (https://www.elastic.co/guide/en/kibana/7.6/managing-licenses.html#update-license). Did you experience any issues with that approach? What is the version of Kibana and Elasticsearch you are running with ECK?

8

Thats the problem, the Kibana license management UI gives a permission error, all the GET, DELETE _license gives no output and an error with curl.

elastic server is still running bu no reponse on port 9200 (see the errors in the beginning of this post)

version 7.4.2

9

That's very odd. Only health and stats API calls should be blocked once the licence expires (https://www.elastic.co/guide/en/kibana/7.6/managing-licenses.html#expiration-security). If the node does not respond to any call at all, something else must be wrong. Can you see any errors (apart from the security exceptions) in Elasticsearch logs or the operator log?

10

yes very odd, restart seems to crash kibana aswell with the errors:

│ {"type":"log","@timestamp":"2020-03-25T13:18:09Z","tags":["reporting","esqueue","queue-worker","error"],"pid":8,"message":"k87cp85g0008d146496zv7or - job querying failed: Error: Request Timeout after 30000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:397:9\n    at Timeout.<anonymous> (/usr/share/kib │
│ {"type":"log","@timestamp":"2020-03-25T13:18:33Z","tags":["license","warning","xpack"],"pid":8,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms"}

elastic server

 elasticsearch "at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]",                                                                                                                                                                                                                                           │
│ elasticsearch "at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]",                                                                                                                                                                                                                                                          │
│ elasticsearch "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                            │
│ elasticsearch "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",                                                                                                                                                                                                                                   │
│ elasticsearch "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",                                                                                                                                                                                                                                   │
│ elasticsearch "at java.lang.Thread.run(Thread.java:830) [?:?]"] }                                                                                                                                                                                                                                                                           │
│ elasticsearch {"type": "server", "timestamp": "2020-03-25T13:20:22,905Z", "level": "ERROR", "component": "o.e.x.s.a.f.SecurityActionFilter", "cluster.name": "elastic", "node.name": "elastic-es-default-0", "message": "blocking [cluster:monitor/nodes/stats] operation due to expired license. Cluster health, cluster stats and indices │
│ elasticsearch {"type": "server", "timestamp": "2020-03-25T13:20:27,912Z", "level": "ERROR", "component": "o.e.x.s.a.f.SecurityActionFilter", "cluster.name": "elastic", "node.name": "elastic-es-default-0", "message": "blocking [cluster:monitor/nodes/stats] operation due to expired license. Cluster health, cluster stats and indices │
│ elasticsearch {"type": "server", "timestamp": "2020-03-25T13:20:29,258Z", "level": "ERROR", "component": "o.e.x.s.a.f.SecurityActionFilter", "cluster.name": "elastic", "node.name": "elastic-es-default-0", "message": "blocking [cluster:monitor/stats] operation due to expired license. Cluster health, cluster stats and indices stats │
│ elasticsearch {"type": "server", "timestamp": "2020-03-25T13:20:29,259Z", "level": "ERROR", "component": "o.e.x.m.c.c.ClusterStatsCollector", "cluster.name": "elastic", "node.name": "elastic-es-default-0", "message": "collector [cluster_stats] failed to collect data", "cluster.uuid": "kWqCKAQ-R4-nL-QiR3x8Pg", "node.id": "FLDpzpj3 │
│ elasticsearch "stacktrace": ["org.elasticsearch.ElasticsearchSecurityException: current license is non-compliant for [security]",                                                                                                                                                                                                           │
│ elasticsearch "at org.elasticsearch.license.LicenseUtils.newComplianceException(LicenseUtils.java:27) ~[?:?]",                                                                                                                                                                                                                              │
│ elasticsearch "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:79) ~[?:?]",                                                                                                                                                                                                          │
│ elasticsearch "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                  │
│ elasticsearch "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                                     │
│ elasticsearch "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                                      │
│ elasticsearch "at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                                            │
│ elasticsearch "at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                                                 │
│ elasticsearch "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                                       │
│ elasticsearch "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:385) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                                       │
│ elasticsearch "at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:679) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                          │
│ elasticsearch "at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:45) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                                    │
│ elasticsearch "at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:59) ~[elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                                        │
│ elasticsearch "at org.elasticsearch.xpack.monitoring.collector.cluster.ClusterStatsCollector.lambda$doCollect$0(ClusterStatsCollector.java:95) ~[x-pack-monitoring-7.4.2.jar:7.4.2]",                                                                                                                                                       │
│ elasticsearch "at org.elasticsearch.xpack.monitoring.collector.cluster.ClusterStatsCollector.doCollect(ClusterStatsCollector.java:99) ~[x-pack-monitoring-7.4.2.jar:7.4.2]",                                                                                                                                                                │
│ elasticsearch "at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:88) [x-pack-monitoring-7.4.2.jar:7.4.2]",                                                                                                                                                                                                   │
│ elasticsearch "at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.2.jar:7.4.2]",                                                                                                                                                                        │
│ elasticsearch "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                                                 │
│ elasticsearch "at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]",                                                                                                                                                                                                                                           │
│ elasticsearch "at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]",                                                                                                                                                                                                                                                          │
│ elasticsearch "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.2.jar:7.4.2]",                                                                                                                                                                            │
│ elasticsearch "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",                                                                                                                                                                                                                                   │
│ elasticsearch "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",                                                                                                                                                                                                                                   │
│ elasticsearch "at java.lang.Thread.run(Thread.java:830) [?:?]"] }                                                                                                                             ```

only errors concerning the license
11

I am baffled by why your Elasticsearch node is not responding at all. I tried to re-create your issue by creating a cluster, starting a trial, and then deleting the licence. I did see an exception similar to yours in the log but issuing the following curl request still returned an error message:

curl -k -XGET -u "elastic:$(kubectl get secret quickstart-es-elastic-user -o=go-template='{{ .data.elastic | base64decode}}')" 'https://localhost:9200/_cat/health'

{"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":403}

Have you tried sending requests to other nodes in the cluster to see if the same thing happens?

12

I do get some repons now

[root@elastic-es-default-0 elasticsearch]# curl -k -XGET -u elastic:xxx https://localhost:9200/_cat/health
{"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":403}
[root@elastic-es-default-0 elasticsearch]# curl -k -XGET -u elastic:xxx https://localhost:9200/_license   
{
  "license" : {
    "status" : "expired",
    "uid" : "ed321833-5ff0-4e85-97b9-9d22e6ab3a72",
    "type" : "trial",
    "issue_date" : "2020-02-13T10:40:23.619Z",
    "issue_date_in_millis" : 1581590423619,
    "expiry_date" : "2020-03-14T10:40:23.619Z",
    "expiry_date_in_millis" : 1584182423619,
    "max_nodes" : 1000,
    "issued_to" : "elastic",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}
13

If the nodes are responding, you should be able to revert to basic by issuing the start_basic API call. https://www.elastic.co/guide/en/elasticsearch/reference/7.6/start-basic.html

14 
[root@elastic-es-default-0 elasticsearch]# curl -k -XDELETE -u elastic:xxx https://localhost:9200/_license
{"acknowledged":true}[root@elastic-es-default-0 elasticsearch]# 
[root@elastic-es-default-0 elasticsearch]# curl -k -XGET -u elastic:xxx https://localhost:9200/_license
{ }

I'm able to remove the license but a restart off the pods keeps giving the same license errors, anythin else I will have to remove?

15

ah yes fixed, thank you very much :slight_smile:

[root@elastic-es-default-0 elasticsearch]# curl -k  -X POST -u elastic:xxx https://localhost:9200/_license/start_basic
{"acknowledged":true,"basic_was_started":true}[root@elastic-es-default-0 elasticsearch]# 
[root@elastic-es-default-0 elasticsearch]# curl -k -XGET -u elastic:xxx https://localhost:9200/_license
{
  "license" : {
    "status" : "active",
    "uid" : "333da05b-0961-414e-82c0-5e1e7f6484f0",
    "type" : "basic",
    "issue_date" : "2020-03-25T15:12:33.513Z",
    "issue_date_in_millis" : 1585149153513,
    "max_nodes" : 1000,
    "issued_to" : "elastic",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}