Endpoint _cat/indices doesn't work when "license expired"?

[Mathemaphysics]

One day, no countdown, I was told after adding data to my ES server through kibana that I can't view indices.

What do I have to do just to use the _cat/indices endpoint? This is an incredible mess I'm in because of this right now, so if someone could help I would really appreciate it.

I'm using security enabled. But there was no indication that tomorrow I wouldn't have a server to use anymore.

Do I need to just disable security? I've had strange things happens with these containers and I've been slowly moving away from ES because of these issues. Day to day it will claim the license is expired and then it will work normally.

[stephenb (Stephen Brown)]

Hi @Mathemaphysics

What version of Elasticsearch?

What license level expired?

EDIT / UPDATED : Per information / results below

An expired license should only block the additional commercial features, but there are some cluster management and stats functions that are blocked such as that API.

Security Auth/ Auth is part of Basic / Free license since 6.8 / 7.1 BUT there are some features that are blocked with an Expired License that can be restored by applying a Basic License .. see below for how to do that.

[DavidTurner]

@stephenb

An expired licenses do not block that endpoint/API.

Basic indexing and searching should still work with an expired license (so the site doesn't completely fall to bits) but by design management endpoints like GET _cat/indices should indeed not work with an expired license.

It's surprising that there was no warning however, normally there are logs about nearly-expired licenses for quite some time before the expiry date. It's doubly surprising that this seems to vary day-to-day, that sounds like maybe you have two independent clusters with different licenses? Something's wrong with how this system is configured or orchestrated for sure, this is not the usual user experience at all.

[stephenb (Stephen Brown)]

@DavidTurner

Apologies...Wow I stand moderately corrected

Reverting to a Basic license should re-renable That functionality correct

And From our own current docs here

If your license expires, your subscription level reverts to Basic and you will no longer be able to use Platinum or Enterprise features.

So I am confused .... That would imply that all basic functionality shall work. No mention of basic management functions being blocked.

but here for 7.x

I do see this

Access to the cluster health and stats APIs is critical for monitoring and managing an Elasticsearch cluster.

and

Cluster health, cluster stats, and indices stats operations are blocked.

Once the license expires, calls to the cluster health, cluster stats, and index stats APIs fail with a security_exception and return a 403 HTTP status code.

Somehow I have never seen that before...

Perhaps in 7.x You have to manually revert to a basic license before you can use those APIs?

Would reverting to a basic license restore?

Seems inconsistent / odd to block listing indices ...

@Mathemaphysics

What version are you on?

And HUH interesting I just tested this...

I have a 8.9 cluster that is getting near EO license and in the logs I see so that same behavior seems to be in 8.x even though it is not mentioned in the docs...

If you have a new license, please update it. Otherwise, please reach out to
# your support contact.
# 
# Commercial plugins operate with reduced functionality on license expiration:
# - security
#  - Cluster health, cluster stats and indices stats operations are blocked
#  - All data operations (read and write) continue to work
# - watcher

So the question stands ...

Does reverting to a Basic license re-enable the health, cluster stats and indices stats operations when basic security (Free) is enabled?

[stephenb (Stephen Brown)]

@Mathemaphysics

Further Research and Answering my own question

So I found a cluster (8.8.1) I have that has basic security and an expired license and I INDEED see the behavior that @DavidTurner described.

What I did is apply the basic license like this

$ curl -X POST -k -u elastic https://localhost:9200/_license/start_basic?acknowledge=true
Enter host password for user 'elastic':
{"acknowledged":true,"basic_was_started":true}hyperion:docker sbrown$ 

and then the management function started to work again.

$curl -k -u elastic https://localhost:9200/_cat/indices
Enter host password for user 'elastic':
green open .internal.alerts-observability.metrics.alerts-default-000001 VdnxouLrRSOq0Ynf8MEvlg 1 0 0 0 247b 247b
green open .internal.alerts-observability.logs.alerts-default-000001    wTvHnjU-RY254byg5Vu6xg 1 0 0 0 247b 247b
green open .internal.alerts-observability.uptime.alerts-default-000001  hbxTXL39SdGeAyNp2Q-8rg 1 0 0 0 247b 247b
green open .fleet-file-data-agent-000001                                iSgisiiWStSkR4j1I735rw 1 0 0 0 247b 247b
green open .fleet-files-agent-000001                                    uPlGT7HkSH-M3dAELcDWxA 1 0 0 0 247b 247b
green open .internal.alerts-security.alerts-default-000001              LdHq9jKCT-WjLcGUrt6l6Q 1 0 0 0 247b 247b
green open .internal.alerts-observability.slo.alerts-default-000001     bTNc2mAxTOKL4H1bsN5dkQ 1 0 0 0 247b 247b

curl -k -u elastic https://localhost:9200/_cat/health
Enter host password for user 'elastic':
1694274138 15:42:18 elasticsearch green 1 1 26 26 0 0 0 0 - 100.0%

Hope this helps!

Interesting....

Today I Learned