ElasticSearch Error When Integrating with Azure AD for SSO Capabilities

Kehinde_Owens · March 18, 2019, 5:12pm

Hello,

I am currently attempting to use my Azure Active Directory as with my ElasticSearch/Kibana instance in order to give my users SSO capabilities. I receive a 502 error message once I enable the settings in Kibana and ElasticSearch. It should be noted that the ElasticSearch settings have been set in all 3 nodes and I am running Kibana and ElasticSearch in Azure behind a proxy server. Below are the settings I have in my ElasticSearch Nodes:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.authc.token.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
#
xpack.security.authc.realms.samll:
 type: saml
 order: 1
 idp.metadata.path: "https://login.microsoftonline.com/f14d56e6-7812-4709-8bd1-492ff77e424d/federationmetadata/2007-06/federationmetadata.xml?appid=2bc165f7-924f-4fba-ac87-de813d2f4ee0"
 idp.entity_id: "https://sts.windows.net/f14d56e6-7812-4709-8bd1-492ff77e424d/"
 sp.entity_id: "https://investc-policydashboard.drlteam.net/"
 sp.acs: "https://investc-policydashboard.drlteam.net/api/security/v1/saml"
 sp.logout: "https://login.microsoftonline.com/common/wsfederation?wa=wsignout1.0"
 attributes.principal: "nameid:persistent"

And my Kibana Settings looks as follows:

#Xpack Settings for Application Gateway
xpack.security.public:
 protocol: https
 hostname: investc-policydashboard.drlteam.net
 port: 443

ikakavas · March 19, 2019, 2:20pm

Hi

Please use </> or backtics (```) in order to format your messages. It's really hard to read through and to know whether there is a mistake in your settings or in the format. You can use the preview panel on the right to see how your message looks like before you post it.

Which component is returning this 502 and when, after what ? Please share some more details.

One thing that stands out for sure is that since you are using a proxy, you need to use the port number in the sp.acs even if it is the default for the protocol you use ( https ) . See this issue if you need more details.

We have a very detailed blog post on configuring SAML with Azure AD, have you gone through it ?

Kehinde_Owens · March 19, 2019, 2:49pm

Sorry for the lack of info, i've edited the post to make it easier to read. To answer your question though, I receive a 502 error once I enable the xpack settings in the Kibana.yml file which leads me to believe this is where the issue is occurring. The backend seems to be working fine it's only when attempting to access my Kibana URL using the associated IP address.

You mentioned sp.acs should include the port number (which is 443) i went through an article and this wasn't mentioned. I'll go through the one you attached and see if I can make any changes. I'll post here with updates. Thanks

ikakavas · March 19, 2019, 3:00pm

Apologies, but this is still not very clear to me. Maybe I asked the wrong questions.

Where do you have Kibana and Elasticsearch installed ? Is this on premise ? Is it Elastic Cloud ? Somewhere else ?

What does "enabling the xpack settings" entail ?

Please note that YAML is whitespace sensitive.

#Xpack Settings for Application Gateway
xpack.security.public:
protocol: https
hostname: investc-policydashboard.drlteam.net
port: 443

is not the same as

#Xpack Settings for Application Gateway
xpack.security.public:
  protocol: https
  hostname: investc-policydashboard.drlteam.net
  port: 443

and this is part of the reason that correctly formatting your questions, makes it easier for people to assist. Can you clarify how your settings look like ? You can copy paste the whole block within triple backticks , for example

  ```
  this: is
    your: yaml
    config: "correctly formatted"
  ```

Which article would that be ? We'd be more than happy to make an adjustment so that other people can benefit from it.

Kehinde_Owens · March 19, 2019, 3:31pm

I am using Azure Cloud to host these applications. I have Kibana on a VM (Linux) and ElasticSearch on 3 separate nodes (Linux as well).
You are right, I need to be more specific with how I write my code, I will edit it and be more specific as to how it's formatted.
The article(s) I used for this task are https://www.elastic.co/blog/saml-based-single-sign-on-with-elasticsearch-and-azure-active-directory

https://www.elastic.co/guide/en/x-pack/current/saml-guide.html

Also, I noticed in your issue that you mention /api/v1/saml should be used. In the article I used it stated that /api/security/v1/saml should be used.. which is correct?

Give me 10-20 minutes to re-upload the correctly formatted code. Thanks

(Update) - It's been updated

Kehinde_Owens · March 19, 2019, 6:40pm

Update - Attempted it again by using the following settings in my elasticsearch.yml:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.authc.token.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
#
xpack.security.authc.realms.samll:
 type: saml
 order: 1
 idp.metadata.path: "https://login.microsoftonline.com/DirectoryID/federationmetadata/2007-06/federationmetadata.xml?appid=AppID"
 idp.entity_id: "https://sts.windows.net/DirectoryID/"
 sp.entity_id: "https://investc-policydashboard.drlteam.net/"
 sp.acs: "https://investc-policydashboard.drlteam.net:443/api/security/v1/saml"
 sp.logout: "https://login.microsoftonline.com/common/wsfederation?wa=wsignout1.0"
 attributes.principal: "nameid:persistent"

And in my Kibana.yml:
xpack.security.authProviders: [saml, basic]
server.xsrf.whitelist: [/api/security/v1/saml]
#Xpack Settings for Application Gateway
xpack.security.public:
 protocol: https
 hostname: investc-policydashboard.drlteam.net
 port: 443

Not much is different except for the :443 added in the sp.acs. After adding these settings I still received a 502 error. What's worse, is when I tried to revert my changes the 502 error remained and Kibana now has these logs when checking the status of it:

Mar 19 18:28:38 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:38Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"Unable to revive connection: http://10.20.1.4:9200/"}
Mar 19 18:28:38 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:38Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"No living connections"}
Mar 19 18:28:39 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:39Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"Unable to revive connection: http://10.20.1.4:9200/"}
Mar 19 18:28:39 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:39Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"No living connections"}
Mar 19 18:28:39 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:39Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"Unable to revive connection: http://10.20.1.4:9200/"}
Mar 19 18:28:39 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:39Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"No living connections"}
Mar 19 18:28:41 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:41Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"Unable to revive connection: http://10.20.1.4:9200/"}
Mar 19 18:28:41 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:41Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"No living connections"}
Mar 19 18:28:43 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:43Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"Unable to revive connection: http://10.20.1.4:9200/"}
Mar 19 18:28:43 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:43Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"No living connections"}

This was not the case before I applied the changes so I know it isn't some other configuration that i'm missing.

Any help would be appreciated. Thanks

ikakavas · March 20, 2019, 7:09am

Mar 19 18:28:38 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:38Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"Unable to revive connection: http://10.20.1.4:9200/"}
Mar 19 18:28:38 drl-kibana kibana[4072]: {"type":"log","@timestamp":"2019-03-19T18:28:38Z","tags":["warning","elasticsearch","admin"],"pid":4072,"message":"No living connections"}

means that Kibana cannot connect to Elasticsearch.

I think I spotted your issue:

 idp.metadata.path: "https://login.microsoftonline.com/DirectoryID/federationmetadata/2007-06/federationmetadata.xml?appid=AppID"

this is definitely wrong. You need to specify your AppID there. Elasticsearch will attempt to fetch the SAML Metadata from the https://login.microsoftonline.com/DirectoryID/federationmetadata/2007-06/federationmetadata.xml?appid=AppID URL and as you can see for yourself, going there with your browser, this returns an error :

 Request Id: 328304e4-ae4b-48b2-80d5-7518beac1900
Correlation Id: 7693f5a6-0be8-4c17-8ba2-db350a506409
Timestamp: 2019-03-20T06:54:55Z
Message: AADSTS90023: The appid must be a GUID.
Advanced diagnostics: Enable
If you plan on getting support for an issue, turn this on and try to reproduce the error. This will collect additional information that will help troubleshoot the issue.

Elasticsearch stops because it can't load the metadata and Kibana can't connect to it. You need to fix this part in your Elasticsearch configuration , see the section named " Application Manifest and appRoles" in our blog, as it talks about the Application ID and where to find it.

Port 443 :

This article doesn't mention 443 as the setup there uses a different port ( 5601 ) for http over TLS. The GH issue has all the details about why 443 is needed for now and in which cases. Let me know if you have more questions about this.

API Path

/api/security/v1/saml is the correct one as seen in our documentation and blog posts. Thanks for spotting the error in the GH issue, I've updated the issue description

502

I'm still not sure what "gives you a 502 once you enable the xpack settings in the Kibana.yml file" . Maybe this will become irrelevant when you fix the metadata url, but please do clarify this further if you get additional errors. Is it Kibana that gives you this error? Azure settings? something else ? A concrete example saying "I navigate to the X URL" or "I click this button in that section of that UI" would be much helpful.

Hope this helps

Kehinde_Owens · March 20, 2019, 11:37am

Thanks for the reply but I don't think that's the issue, I have the AppID and the DirectoryID in the .yaml file I just removed it from this thread for security purposes. I will add them back to the thread as I suppose there isn't much risk of putting them in this thread.

To further elaborate on your 502 point. I make all the changes needed in the nodes for Saml, restart the services, and everything remains working. But when I enable the settings in Kibana (the settings can be seen above), restart the service, and then I attempt to navigate to https://investc-policydashhboard.drlteam.net is when I see the 502 error.

ikakavas · March 20, 2019, 11:56am

Can you share the (at least the relevant parts of your) logs from the proxy, Kibana and elasticsearch ? Since you get a 502 trying to access Kibana which lies behind a proxy, we need to figure out:

Does Elastcsearch start successfully ?
Does Kibana start successfully ?
Can Kibana communicate with Elasticsearch ?
Is the 502 thrown by your proxy or by Kibana ?

We focused on the SAML configuration but it can be something unrelated to that

Kehinde_Owens · March 20, 2019, 12:22pm

Will do, bogged down at the moment; will have it uploaded by COB.

ikakavas · March 20, 2019, 12:29pm

no problem , there are no SLAs here in the forums, just best effort form the community - and this goes both ways

Kehinde_Owens · March 21, 2019, 1:50pm

Kibana started successfully

Pic 5.png2104×720 629 KB

Clearer look at my Kibana settings

Pic 7.png784×270 77 KB
ElasticSearch started successfully

pic 3.png2308×474 485 KB

Clearer look of my settings

Pic 2.png2302×534 406 KB
This is the error I get when attempting to communicate (curl) with ElasticSearch after both instances have been set up for SAML

curl "http://10.20.1.10:9200"
curl: (52) Empty reply from server

The 502 is thrown by the proxy. When checking the status of Kibana on the backend it appears to be running fine as you can see in the screenshots above.

What I can say is that HTTPS has been assigned to Kibana on the network layer but not on the master node for Elasticsearch (which has a public facing IP address). Not sure if this could be an issue.

ikakavas · March 21, 2019, 3:42pm

Hi there

Please don't post images of text as they are hard to read, may not display correctly for everyone, and not searchable.

What I can see from your status outputs is that both Elasticsearch and Kibana didn't suffer from any unrecoverable errors on startup, not that they are running successfully. Kibana's output clearly contains parts of a stacktrace that indicates an exception. The logs will be much more interesting.

The elastic stack supports the SAML 2.0 Web Browser Single Sign on profile, and as such you are supposed to use a browser to authenticate via SAML and you need to go through Kibana. You can't use the Elasticsearch REST API directly. Our blog post contains an introduction to SAML and how the authentication flow happens, this might be helpful !

What are you trying to access when you get this 502 ? Kibana or Elasticsearch ? Sorry for all these questions but for the rest of us, this might not be as obvious as it is for you that you experiencing the issue.

t HTTPS has been assigned to Kibana on the network layer

I'm not really sure what that means actually. Is it just that you have enabled https for Kibana ?

In your config, I can see that TLS is enabled for both the http and the network layer, so can you explain what the above means?

I believe the truth of what happens, lies in the Kibana and Elasticsearch logs so sharing them would be the next step to resolving your issue

system · April 18, 2019, 3:42pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.