Enabling xpack and binding kibana to a non loopback ip makes it unreachable from another workspace

Elastic Stack Elasticsearch
1

I'm following a tutorial you provide as part of your "learn at home" initiative called "Fundamentals of Securing Elasticsearch". I'm actually following along in my own environment. I have a AWS workspace setup and one EC2 instance running ELK.

I have so far done the following successfully...

  • Configured elasticsearch and kibana for their default ports and localhost
  • Enabled xpack.security.enabled: true
  • Configured for single node discovery.type: single-node
  • Setup built-in uses w/ bin/elasticsearch-setup-passwords interactive
  • Added the built-in kibana user to the kibana keystore
  • Verified security was working by issuing a curl request without credentials which caused a security_exception as it should
  • Verified elastic user was working issuing a curl request with it's credentials which worked perfectly

My two configuration files at this point look like the following...

Kibana.yml

server.port: 5601
server.host: "127.0.0.1"
elasticsearch.hosts: ["http://127.0.0.1:9200"]

Elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 127.0.0.1
http.port: 9200
discovery.type: "single-node"
xpack.security.enabled: true

At this point I wanted to test the Kibana log-in page with the elastic user. However, I wanted to use a brower from my AWS workspace to access Kibana. The only setting I changed to test this out was server.host: 127.0.0.1 in the Kibana config to be server.host: "172.32.xx.xx" where 172.32.xx.xx is my EC2 IP. All other settings in the kibana and elasticsearch configs stayed the same for this test.

I restarted both services then opened up a browser in my AWS workspace and navigated to http://172.32.xx.xx:5601. The webpage just displayed a blank white screen and at the bottom indicated a "Waiting for 172.32.xx.xx" status. I waited for 5 minutes but nothing happened.

I noticed my EC2 instance was kind of hung so I had to restart my EC2 instance which was weird. After the EC2 restarted I thought I would try disabling xpack xpack.security.enabled: false and then see what would happen by trying to access Kibana from my browser again in my AWS workspace. This time it worked and the kibana home screen appeared.

Why can't I bind Kibana to my EC2 IP server.host: "172.32.xx.xx" with xpack enabled and access the Kibana login page from a browser in my AWS workspace?

2

Hi,
could it may be a firewall issue?
Can you try to curl kibana from the EC2 Instance kibana is running on?
use this command:

curl 172.32.xx.xx:5601 -L

Please use the -L parameter, otherwise you will get a empty response.
and do the same from your workspace.

Also can you have a look at the kibana logs?

1 Like
3

@KoettingSimon

There's quite alot of output from issuing that command. I'll paste some of it in...

<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"/><meta name="viewport" content="width=device-width"/><title>Kibana</title><style>
        @font-face {
          font-family: 'Inter UI';
          font-style: normal;
          font-weight: 100;
          src: url('/ui/fonts/inter_ui/Inter-UI-Thin-BETA.woff2') format('woff2'), url('/ui/fonts/inter_ui/Inter-UI-Thin-BETA.woff') format('woff');
        }
...
[security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate=\&quot;Basic realm=\\\&quot;security\\\&quot; charset=\\\&quot;
...
<div class="kibanaWelcomeText" data-error-message="Kibana did not load properly. Check the server output for more information.">Loading Kibana</div></div><div class="kibanaWelcomeView" id="kbn_legacy_browser_error" style="display:none"><div class="kibanaLoaderWrap"><div class="kibanaWelcomeLogoCircle"><div class="kibanaWelcomeLogo"></div></div></div><h2 class="kibanaWelcomeTitle">Please upgrade your browser</h2><div class="kibanaWelcomeText">This Kibana installation has strict security requirements enabled that your current browser does not meet.</div></div><script>
            // Since this is an unsafe inline script, this code will not run
            // in browsers that support content security policy(CSP). This is
            // intentional as we check for the existence of __kbnCspNotEnforced__ in
            // bootstrap.
            window.__kbnCspNotEnforced__ = true;
          </script><script src="/bundles/app/login/bootstrap.js"></script></body></html>

Here is what I see in /var/log/messages after issuing the curl command you said to run.

14895 May  6 05:54:39 ip-172-32-56-218 kibana: {"type":"response","@timestamp":"2020-05-06T05:54:39Z","tags":[],"pid":3492,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"curl/7.29.0","host":"172.32.56.218:5601","accept":"*/*"},"remoteAddress":"172.32.56.218","userAgent":"172.32.56.218      "},"res":{"statusCode":302,"responseTime":15,"contentLength":9},"message":"GET / 302 15ms - 9.0B"}
14896 May  6 05:54:39 ip-172-32-56-218 kibana: {"type":"response","@timestamp":"2020-05-06T05:54:39Z","tags":[],"pid":3492,"method":"get","statusCode":200,"req":{"url":"/login?next=%2F","method":"get","headers":{"user-agent":"curl/7.29.0","host":"172.32.56.218:5601","accept":"*/*"},"remoteAddress":"172.32.56.218","userAgent":      "172.32.56.218"},"res":{"statusCode":200,"responseTime":55,"contentLength":9},"message":"GET /login?next=%2F 200 55ms - 9.0B"}
4

ok, this is executed from the EC2 instance where kibana is running?
If yes, execute the same command on your workspace so we can check if its a kibana problem or a network (firewall) related problem that your workstation has no access to the EC2 instance on this port.

1 Like
5

@KoettingSimon

Yes, the above output is from executing the curl command on the EC2 instance where Kibana and Elasticsearch are running.

Below is the output when executing the curl command on my AWS workspace...

<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"/><meta name="viewport" content="width=device-width"/><title>Kibana</title><style>
        @font-face {
          font-family: 'Inter UI';
          font-style: normal;
          font-weight: 100;
          src: url('/ui/fonts/inter_ui/Inter-UI-Thin-BETA.woff2') format('woff2'), url('/ui/fonts/inter_ui/Inter-UI-Thin-BETA.woff') format('woff');
        }
...
[security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate=\&quot;Basic realm=\\\&quot;security\\\&quot;
...
<div class="kibanaWelcomeLogo"></div></div></div><div class="kibanaWelcomeText" data-error-message="Kibana did not load properly. Check the server output for more information.">Loading Kibana</div></div><div class="kibanaWelcomeView" id="kbn_legacy_browser_error" style="display:none"><div class="kibanaLoaderWrap"><div class="kibanaWelcomeLogoCircle"><div class="kibanaWelcomeLogo"></div></div></div><h2 class="kibanaWelcomeTitle">Please upgrade your browser</h2><div class="kibanaWelcomeText">This Kibana installation has strict security requirements enabled that your current browser does not meet.</div></div><script>
            // Since this is an unsafe inline script, this code will not run
            // in browsers that support content security policy(CSP). This is
            // intentional as we check for the existence of __kbnCspNotEnforced__ in
            // bootstrap.
            window.__kbnCspNotEnforced__ = true;
          </script><script src="/bundles/app/login/bootstrap.js"></script></body></html>

...and trailing the /var/log/messages again showed...

14906 May  6 06:11:48 ip-172-32-56-218 kibana: {"type":"response","@timestamp":"2020-05-06T06:11:48Z","tags":[],"pid":3492,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"172.32.56.218:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"172.33.2.22","userAgent":"172.33.2.22"},"      res":{"statusCode":302,"responseTime":3,"contentLength":9},"message":"GET / 302 3ms - 9.0B"}
14907 May  6 06:11:48 ip-172-32-56-218 kibana: {"type":"response","@timestamp":"2020-05-06T06:11:48Z","tags":[],"pid":3492,"method":"get","statusCode":200,"req":{"url":"/login?next=%2F","method":"get","headers":{"host":"172.32.56.218:5601","user-agent":"curl/7.61.1","accept":"*/*"},"remoteAddress":"172.33.2.22","userAgent":"1      72.33.2.22"},"res":{"statusCode":200,"responseTime":19,"contentLength":9},"message":"GET /login?next=%2F 200 19ms - 9.0B"}
6

Ok, so in this case its no firewall problem.
Please share more of the kibana log, best would be restart kibana and post all the logs from this restart.

1 Like
7

@KoettingSimon

I restarted via sudo systemctl restart kibana.service. The logs from var/log/messages exceeded the limit so I threw it up on a public repo here.

8

Are this all logs? This logs looks ok.
Are there any errors later? Or if you try to access kibana?

1 Like
9

@KoettingSimon

I'm lost for words. It's working now as I can get to the login page and log in. That is the strangest thing ever. I literally have no idea what I did. Maybe my EC2 was in a weird state?! I did just turn on verbose logging but since it's working not sure what the need is to even look at it now.

I'm not sure what to say but thanks for sticking with me at 1:16AM in the morning. I appreciate the help.

Cheers ~ S

10

@syost, great to hear, all i can say about it is:

grafik
grafik1400×800 138 KB

:smiley:

Cheers,
Simon

2 Likes
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.