Configuring and debugging anonmyous access to Elasticsearch

Hi everyone,

I'm trying to configure our ES instance so that anonymous users can index documents (but not modify existing docs).

Here's what I did:

  • Enabled security xpack.security.enabled: true
  • Configured a custom role in the UI. I tried various options (set "run as" to _anonymous, use all as the indices privilege to be sure that wasn't the problem) but non worked.
  • Use this custom role by adding the following in elasticsearch.yml (and restarting ES):
    xpack.security.authc:
     anonymous:
       roles: xwiki_anonymous
    

When I try to connect from a client using the Java Client API, I get a 403:

2022-06-01 13:21:51,294 [Active Installs 2 Ping Thread] DEBUG o.e.c.RestClient               - request [PUT https://xxx/activeinstalls2/_ingest/pipeline/set-timestamp] returned [HTTP/1.1 403 Forbidden] 
2022-06-01 13:21:51,300 [Active Installs 2 Ping Thread] TRACE tracer                         - curl -iX PUT 'https://xxx/activeinstalls2/_ingest/pipeline/set-timestamp' -d '{"description":"Set current date to be the server date/time and fill first ping date if empty","processors":[{"set":{"field":"date.current","value":"{{{_ingest.timestamp}}}"}},{"set":{"field":"date.first","override":false,"value":"{{{date.current}}}"}}]}'
# HTTP/1.1 403 Forbidden
# Date: Wed, 01 Jun 2022 11:21:51 GMT
# Content-Type: text/html
# Transfer-Encoding: chunked
# Connection: keep-alive
# CF-Cache-Status: DYNAMIC
# Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
# Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=yyy"}],"group":"cf-nel","max_age":604800}
# NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
# Server: cloudflare
# CF-RAY: 7147916faa0f40e7-CDG
#
# <html>
# <head><title>403 Forbidden</title></head>
# <body>
# <center><h1>403 Forbidden</h1></center>
# <hr><center>nginx/1.18.0</center>
# </body>
# </html>

Any idea what I'm doing wrong?

Are there options to debug this? I googled and couldn't find any doc to help debugging role-based access.

Thanks a lot!

Create or update ingest pipeline requires manage_pipeline cluster privilege (docs)

I'm trying to configure our ES instance so that anonymous users can index documents (but not modify existing docs).

If this is your goal, it is not clear to me why you tested with creating/updating pipelines. The index privilege for this is create_doc (docs)

Also the error message you shared is not directly from Elasticsearch. Direct response from Elasticsearch would be more helpful if you run into further issues.

Thanks. I have tried with all for both cluster and index privileges and I get the same error.

The client does several things: set some ingest pipeline + index some documents.

I guess this is what I'm asking. How do I debug permissions issues in ES? Right now, what I've pasted is the result of the following logging configuration:

<logger name="org.elasticsearch.client" level="trace"/>
<logger name="org.elasticsearch.client.sniffer" level="trace"/>
<logger name="tracer" level="trace"/>

What else can I do to help diagnose the problem?

Thanks a lot for your help!

PS: If I disable security all is working fine.

You'll need either server logs or server response, not client logs. It's probably better to use a simple command tool like curl for direct server response.

I have tried with all for both cluster and index privileges and I get the same error.

If you call GET _security/_authenticate with the anonymous user, what response do you get?

Hi @Yang_Wang . Sorry I was on holidays and I'm back only today. Thx for your last message.

I had already checked the server logs at /var/log/elasticsearch/elasticsearch.log but I don't see anything special (request results are not logged there apparently). Maybe I'm checking the wrong log file?

This will return the same result if called from the client side.

Actually just doing a GET on the root URL of the ES server returns a 502.

$ curl https://xxxx/activeinstalls2/
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
$ curl https://xxx/activeinstalls2/_security/_authenticate
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>

If I turn off security, then this works fine so it must have to do with the security setup.

Thanks!

Sorry but I am not able to assist further unless we have direct response and log (better at debug level) from the ES server instead of the proxy (nginx) response.

With security turned off (xpack.security.enabled: false):

$ curl http://localhost:9200/
{
  "name" : "elk-xwikiorg-prod",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "iiWb4gEYRuuoIQPNI0Wzvw",
  "version" : {
    "number" : "8.2.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "b174af62e8dd9f4ac4d25875e9381ffe2b9282c5",
    "build_date" : "2022-04-20T10:35:10.180408517Z",
    "build_snapshot" : false,
    "lucene_version" : "9.1.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

With security turned on (xpack.security.enabled: true):

$ curl http://localhost:9200/
curl: (52) Empty reply from server

I've put logger.org.elasticsearch: DEBUG in elasticsearch.yml and I get debug logs but nothing related to the HTTP query from curl. Could you provide more information on how to get debug logs for permissions and query processing?

Thanks

Actually, without debug logs, I see the following which could be one problem:

[2022-06-13T14:10:28,042][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [elk-xwikiorg-prod] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:45876}

So actually it works better with either SSL off or by using HTTPS:

$ curl -k https://localhost:9200/
{
  "name" : "elk-xwikiorg-prod",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "iiWb4gEYRuuoIQPNI0Wzvw",
  "version" : {
    "number" : "8.2.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "b174af62e8dd9f4ac4d25875e9381ffe2b9282c5",
    "build_date" : "2022-04-20T10:35:10.180408517Z",
    "build_snapshot" : false,
    "lucene_version" : "9.1.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tag

It's working fine now. Was some nginx configuration issue mixed with other things.

Thanks for the help.