Modify ciphers used on Elasticsearch Service API endpoints?

[glg]

We use Elasticsearch Service hosted on AWS and use the hosted App Search for a custom search feature on our site. We've received a report from an end user that when they submit a search request, that an authentication dialog appears:

The end user's IT department suggested that the SSL configuration violated their policies, but I'm unclear what exactly their security settings did to trigger the basic HTTP auth dialog.

That said, I ran an SSL Server Test (Powered by Qualys SSL Labs) on the App Search endpoint and the results show that the App Search API endpoint does not support forward secrecy and has had its grade "capped to B" as a result:

I don't have enough details yet from the end user's IT department to know if this is specifically what their systems are objecting to, but is it possible to control the ciphers used by the API endpoint so that I could get a passing A+ grade by SSL Labs?

Specifically it looks like only TLS_ECDHE_RSA* is available but TLS_ECDHE_ECDSA* is preferred by SSL Labs.

[Sean_Story]

Hi @glg , sorry you're having this issue.

@glg

We've received a report from an end user that when they submit a search request, that an authentication dialog appears

Can you clarify, what is the user doing to "submit a search request" here, and what service are they hitting? Are they hitting a web app that you've developed that is powered by App Search? Are they attempting to hit the App Search UI in Enterprise Search (older versions) or in Kibana (newer versions)? Are they attempting to hit a deployed Search UI as generated by App Search? Are they attempting to hit the search endpoint of App Search?

App Search is intended to receive search requests using a public search key, so it's odd to me that you user is being presented with Basic Authentication prompts. This is why I'm wondering if they are actually hitting something besides where they would submit a search request to App Search.

The end user's IT department suggested that the SSL configuration violated their policies, but I'm unclear what exactly their security settings did to trigger the basic HTTP auth dialog.

I'm not a browser expert, but this shouldn't be related to SSL settings. Usually the browser will give you these prompts if it receives a 401 error with a header like WWW-Authenticate: Basic. App Search should not be doing that, so I expect your end user is actually hitting something else.

That said, I ran an SSL Server Test (Powered by Qualys SSL Labs) on the App Search endpoint and the results show that the App Search API endpoint does not support forward secrecy and has had its grade "capped to B" as a result

Reading a bit about forward secrecy, I don't expect that is something that we have on our roadmap to support. If you have a support relationship with Elastic, you may file an Enhancement Request for this feature.

I don't have enough details yet from the end user's IT department to know if this is specifically what their systems are objecting to, but is it possible to control the ciphers used by the API endpoint so that I could get a passing A+ grade by SSL Labs?

Enterprise Search uses the cipher suites available to it via the JVM and Open SSL. If you want newer cipher suites, I suggest you update your Java version, your Open SSL, and your Enterprise Search version.

[glg]

Sorry if my post wasn't clear but we're using Elastic's hosted Elasticsearch Service (8.4.1), so JVM/Open SSL etc are beyond my reach, Elastic hosts and manages those bits.

The App Search (aka Enterprise Search) endpoint I'm referring to is what is automatically created by Elastic Cloud and is what we copy from the deployment details. We are using the latest Search UI in headless mode since we do not use React. Our integration takes the user's search input and fires off the search request using headless Search UI (configured with the endpoint and Public Search Key).

This works flawlessly for everyone except for at least one user who is in a restricted health care setting, who instead gets the basic HTTP auth dialog due to something being blocked or manipulated by their IT department's restrictions.

Usually the browser will give you these prompts if it receives a 401 error with a header like WWW-Authenticate: Basic. App Search should not be doing that, so I expect your end user is actually hitting something else.

Ah, the IT department did share a screen shot where they were in fact getting the 401 error in the browser's developer console. Again, I don't know specifically what their IT team has done to lock down browsers, they just provided a brief summary that said:

the site does not pass our SSL inspection controls which we have enforced to monitor PHI being transmitted, thus resulting in the strange prompt for credentials the vendor’s site service with AWS as shown in the screenshot below

This seems to suggest they are somehow blocking Search UI from submitting the Public Search Key, which causes a 401. I just tested this and set the searchKey = '' and this does trigger the 401 but I still cannot reproduce how they are getting the HTTP auth prompt. EDIT: the HTTP auth dialog is shown by Chrome, my Firefox does not do this.

Reading a bit about forward secrecy, I don't expect that is something that we have on our roadmap to support. If you have a support relationship with Elastic, you may file an Enhancement Request for this feature.

Any specific reason why not? It looks like TLS_ECDHE_ECDSA* is supported by JDK but I have no idea how the endpoints are configured when hosted by Elastic on Amazon, other than that the endpoint is using a Let's Encrypt generated certificate. It seems this will be an issue for anyone that is subject to 3rd party audits or doing B2B sites with clients in a health care or other restricted setting. If this is a hard "no" from Elastic, that's unfortunate since that only leaves your customers with options ranging from running a proxy to self-hosting. It would be far preferable to be able to have these endpoints use the latest and greatest cipher suites.

[Sean_Story]

@glg

Sorry if my post wasn't clear but we're using Elastic's hosted Elasticsearch Service (8.4.1), so JVM/Open SSL etc are beyond my reach, Elastic hosts and manages those bits

My mistake! I must've misread your first post, and thought you were saying that you'd deployed just on top of AWS EC2 infrastructure.

You are correct then, the JVM and Open SSL settings are beyond your reach there. Luckily, 8.4.1 is quite recent, so the cipher suites you have available there should be quite up to date. And looking at your screenshot, it seems like cipher strength is in fact "in the green", so perhaps this isn't even a problem for you?

this does trigger the 401 but I still cannot reproduce how they are getting the HTTP auth prompt. EDIT: the HTTP auth dialog is shown by Chrome, my Firefox does not do this.

I've reproduced too, which surprised me. I opened a dialog with the devs who own this part of the code, to see if they think this is expected behavior or a bug, and they confirmed that we do support basic auth for the APIs, so this is expected behavior. What's still unexpected to me is that you/your customers are hitting /api/as/v1/engines/<engine>l/search.json in the browser. Or is it your search experience that is also returning a header such that the basic auth dialog is displayed?

@glg

Reading a bit about forward secrecy, I don't expect that is something that we have on our roadmap to support. If you have a support relationship with Elastic, you may file an Enhancement Request for this feature.

Any specific reason why not? It looks like TLS_ECDHE_ECDSA* is supported by JDK but I have no idea how the endpoints are configured when hosted by Elastic on Amazon, other than that the endpoint is using a Let's Encrypt generated certificate.

I don't know which one of us is misunderstanding here, but I think there is a misunderstanding. From what I understood from your original post, the real problem your customer has is with the "B" rating. And the "B" rating is because of the lack of support for Forward Secrecy. And my brief google of Forward Secrecy has do do with the existence of individual session keys, where as the App Search API utilizes two static keys - a private admin key and a public search key. I don't anticipate our architecture of key usage changing any time soon. But none of this is related to cipher suites. As I said above, the Qualys SSL Labs seems to find the cipher strength "in the green". Am I misunderstanding something?

[glg]

I've not yet heard back from the end user or their IT department and still do not know what specifically their systems are objecting to. When I received the note that "the site does not pass our SSL inspection controls", I simply ran the SSL Labs scan which has been my trusty sidekick over the years to point out potential issues and this was the only thing I could find that SSL Labs objected to. Further down in the resulting report, there is an orange line for Forward Secrecy:

This links to their wiki which states:

You need to support and prefer ECDHE suites in order to enable forward secrecy with modern web browsers. To support a wider range of clients, you should also use DHE suites as fallback after ECDHE. Avoid the RSA key exchange unless absolutely necessary.

In this context, Forward Secrecy has to do with the TLS connection. As far as I'm aware, the public search key is an authentication token of sorts and is not used for encrypting anything, so it has nothing to do with Forward Secrecy and is not what SSL Labs is objecting to. SSL Labs is objecting to the fact that FS is not set up correctly for the TLS connection, and from my read of it this is because the endpoint is not configured with their preferred ciphers as listed in the wiki.

At any rate, we can bracket the IT team's objections and reduce my inquiry to simply "what can be done to make sure that Elasticsearch Service API endpoints get an A+ on an SSL Labs scan?" The answer so far is "nothing" on my end, so that seems to leave this up to whichever internal team handles the configuration automation for AWS deployments.

[Carlos_D (Carlos)]

Hi @glg !

I've been checking our AWS SSL setup with our cloud team:

• We cannot drop support for some ciphers, as some of our customer use them (IE11, I'm looking at you). This makes SSL Labs to drop the grade to B.
• We are aware of this, and provide safe cipher alternatives for being chosen by the vast majority of browsers.

Testing with testssl.sh shows that we support PFS. Please see the attached screenshots.

So basically, we are not going to drop support for the offending ciphers until we have a clear understanding on the impact this provides in our customers and have a way for them not to be impacted. I hope you and the IT team understand the situation.

That said, the issue the user is having seems related to intermediate networking equipment removing the authentication headers needed, as you had already worked out with @Sean_Story . This should be the focus on the investigation. In case there are more assurances needed from the IT team in order for these restrictions to be removed, let us know what we can do on our end.

Hopefully this helps!

[glg]

Thanks for the reply @Carlos_D! Totally hear you on IE11. We no longer support IE11 on our site (which is also hosted on AWS) and participate in Microsoft's IE Compatibility List so Windows will automatically open the site in Edge instead of IE, so we've already dropped the offending ciphers from the AWS ELB SSL settings for the site itself.

At any rate, you've answered my original question: end users of the Elasticsearch Service cannot in any way alter the SSL settings. I'll close this out since whatever is going on with the end user's network environment is beyond our control.