I'm using a functional ELK stack with nginx, but when I install X-Pack and then disable Security with xpack.security.enabled: false in both Kibana and Elasticsearch config files I get "502 Bad Gateway
nginx/1.10.0 (Ubuntu)" when I attempt to access Kibana. If I remove the disabling line I just get the usual Kibana login screen with the message that I can't use Security with a basic license and need to disable it.
Can you check your nginx logs and share some of them here? The logs should tell you what URL the proxy tried to pass through to but was unable to.
Made 5 attempts to access it, and these entries in the nginx logs were the result.
error.log
2016/11/14 08:47:13 [error] 1167#1167: *730 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.23.xxx, server: FQDNS, request: "GET / HTTP/1.1", upstream: "http://[::1]:5601/", host: "FQDNS"
2016/11/14 08:47:21 [error] 1167#1167: *730 no live upstreams while connecting to upstream, client: 10.10.23.xxx, server: FQDNS, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "FQDNS"
2016/11/14 08:47:24 [error] 1167#1167: *730 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.23.xxx, server: FQDNS, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5601/", host: "FQDNS"
2016/11/14 08:47:24 [error] 1167#1167: *730 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.23.xxx, server: FQDNS, request: "GET / HTTP/1.1", upstream: "http://[::1]:5601/", host: "FQDNS"
2016/11/14 08:47:26 [error] 1167#1167: *730 no live upstreams while connecting to upstream, client: 10.10.23.xxx, server: FQDNS, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "FQDNS"
access.log
10.10.23.xxx - username [14/Nov/2016:08:47:13 +0100] "GET / HTTP/1.1" 502 584 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
10.10.23.xxx - username [14/Nov/2016:08:47:21 +0100] "GET / HTTP/1.1" 502 584 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
10.10.23.xxx - username [14/Nov/2016:08:47:24 +0100] "GET / HTTP/1.1" 502 584 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
10.10.23.xxx - username [14/Nov/2016:08:47:26 +0100] "GET / HTTP/1.1" 502 584 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
Strangely the first and next to last entries of error.log seems to be missing pieces. Shows the time and then skips to the end with the host.
Here's an image for completion.
I think the SSL certificate used to serve web traffic on the server being proxied to is not configured properly. Assuming you are creating the certificate files with the openssl command, you get asked a few questions such as:
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:Brooklyn
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Example Brooklyn Company
Organizational Unit Name (eg, section) []:Technology Division
Common Name (e.g. server FQDN or YOUR name) []:examplebrooklyn.com
Email Address []:
See this reference for where I copied that from: https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs
The "Common Name" question is one of the most important, it needs to match a hostname that the host can identify under, or an IP address if you don't have a hostname. When I create self-signed cert for my development setup, I usually use tsullivan.local as the Common Name, as that is a hostname that maps to my localhost and I can actually ping it.
I have to guess that for the Common Name question, you or someone just entered FQDNS at the prompt. The server being proxied to is maybe https://localhost and the SSL cert that comes back says the certificate is valid for FQDNS, which isn't a host the proxy can reach on the network.
Sorry if this is a shot in the dark, but this is just what I think has happened based on the error log entries that say host: "FQDNS".
Oh, that's a mistake on my part. I should've specified "FQDNS" is just a placeholder adress to obfuscate the domain address used. I'm not using SSL currently in any part of the ELK stack.
After some more testing I've discovered that the 5601 port stops being listened to after disabling Security.
Config for nginx:
server {
listen 80;
server_name domain.being.used;
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/htpasswd.users;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
netstat -l, after installing x-pack but before disabling Security:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 :http : LISTEN
tcp 0 0 :ssh : LISTEN
tcp 0 0 localhost:6010 : LISTEN
tcp 0 0 localhost:6011 : LISTEN
tcp 0 0 localhost:5601 : LISTEN
tcp6 0 0 localhost:9200 [::]: LISTEN
tcp6 0 0 localhost:9200 [::]: LISTEN
tcp6 0 0 localhost:9300 [::]:* LISTEN
tcp6 0 0 localhost:9300 [::]:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
tcp6 0 0 localhost:6010 [::]:* LISTEN
tcp6 0 0 localhost:6011 [::]:* LISTEN
localhost:5601 is present.
netstat -l, after disabling Security in both kibana and elasticsearch configs and restarting both services:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 :http : LISTEN
tcp 0 0 :ssh : LISTEN
tcp 0 0 localhost:6010 : LISTEN
tcp 0 0 localhost:6011 : LISTEN
tcp6 0 0 [::]:5000 [::]: LISTEN
tcp6 0 0 localhost:9200 [::]: LISTEN
tcp6 0 0 localhost:9200 [::]:* LISTEN
tcp6 0 0 localhost:9300 [::]:* LISTEN
tcp6 0 0 localhost:9300 [::]:* LISTEN
tcp6 0 0 [::]:5044 [::]:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
tcp6 0 0 localhost:6010 [::]:* LISTEN
tcp6 0 0 localhost:6011 [::]:* LISTEN
tcp6 0 0 localhost:9600 [::]:* LISTEN
localhost:5601 is all of a sudden missing, which would explain why I get bad gateway when trying to connect to Kibana. No idea why this happens, however.
Hm, what do you see in the Kibana logs? Unless there is an error, you should see a message such as:
log [00:49:01.628] [info][listening] Server running at https://tsullivan.local:5601
One thing I should mention is that when you disable Security in Kibana, it has to re-compile the bundles of Javascript code for the removal of Security JS.
I'll share what my logs look like when start Kibana after disabling one of my plugins:
log [17:26:53.978] [info][optimize] Optimizing and caching bundles for graph, kibana, timelion, login, logout and status_page. This may take a few minutes
log [17:28:28.093] [info][optimize] Optimization of bundles for graph, kibana, timelion, login, logout and status_page complete in 94.10 seconds
log [17:28:28.251] [info][status][plugin:kibana@5.0.1] Status changed from uninitialized to green - Ready
log [17:28:28.307] [info][status][plugin:elasticsearch@5.0.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [17:28:28.365] [info][status][plugin:xpack_main@5.0.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [17:28:28.550] [info][status][plugin:graph@5.0.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [17:28:28.567] [info][status][plugin:reporting@5.0.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [17:28:28.606] [info][status][plugin:elasticsearch@5.0.1] Status changed from yellow to green - Kibana index ready
log [17:28:28.638] [info][status][plugin:security@5.0.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [17:28:28.776] [info][license][xpack] Imported license information from Elasticsearch: mode: gold | status: active | expiry date: 2017-02-01T16:59:59-07:00
log [17:28:28.787] [info][status][plugin:xpack_main@5.0.1] Status changed from yellow to green - Ready
log [17:28:28.789] [info][status][plugin:graph@5.0.1] Status changed from yellow to green - Ready
log [17:28:28.798] [info][status][plugin:reporting@5.0.1] Status changed from yellow to green - Ready
log [17:28:28.800] [info][status][plugin:security@5.0.1] Status changed from yellow to green - Ready
log [17:28:28.828] [info][status][plugin:console@5.0.1] Status changed from uninitialized to green - Ready
log [17:28:29.061] [info][status][plugin:timelion@5.0.1] Status changed from uninitialized to green - Ready
log [17:28:29.068] [info][listening] Server running at https://tim-virtual-machine.local:5601
log [17:28:29.069] [info][status][ui settings] Status changed from uninitialized to green - Ready
Note that the optimization took 94 seconds, and the Kibana server doesn't start listening until near the end of the cycle. So the issue could be that you might not be waiting long enough for the optimization process to finish.
{"type":"log","@timestamp":"2016-11-17T07:30:54Z","tags":["status","plugin:monitoring@5.0.0","info"],"pid":15949,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2016-11-17T07:30:54Z","tags":["status","plugin:elasticsearch@5.0.0","info"],"pid":15949,"state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2016-11-17T07:30:54Z","tags":["status","ui settings","info"],"pid":15949,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Elasticsearch plugin is red"}
{"type":"log","@timestamp":"2016-11-17T07:30:54Z","tags":["status","plugin:xpack_main@5.0.0","info"],"pid":15949,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2016-11-17T07:30:54Z","tags":["status","plugin:graph@5.0.0","info"],"pid":15949,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2016-11-17T07:30:54Z","tags":["status","plugin:reporting@5.0.0","info"],"pid":15949,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2016-11-17T07:30:54Z","tags":["status","plugin:security@5.0.0","info"],"pid":15949,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2016-11-17T07:33:24Z","tags":["fatal"],"pid":16685,"level":"fatal","message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","name":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'\n at Error (native)","code":"EACCES"}}
{"type":"log","@timestamp":"2016-11-17T07:33:30Z","tags":["fatal"],"pid":16704,"level":"fatal","message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","name":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'\n at Error (native)","code":"EACCES"}}
{"type":"log","@timestamp":"2016-11-17T07:33:35Z","tags":["fatal"],"pid":16721,"level":"fatal","message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","name":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'\n at Error (native)","code":"EACCES"}}
*** more of the same, until I did a chown on the file ***
{"type":"log","@timestamp":"2016-11-17T07:41:07Z","tags":["fatal"],"pid":18419,"level":"fatal","message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","name":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'\n at Error (native)","code":"EACCES"}}
{"type":"log","@timestamp":"2016-11-17T07:41:12Z","tags":["info","optimize"],"pid":18437,"message":"Optimizing and caching bundles for monitoring, kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2016-11-17T07:45:04Z","tags":["info","optimize"],"pid":18437,"message":"Optimization of bundles for monitoring, kibana, timelion and status_page complete in 231.19 seconds"}
{"type":"log","@timestamp":"2016-11-17T07:45:04Z","tags":["status","plugin:kibana@5.0.0","info"],"pid":18437,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-11-17T07:45:05Z","tags":["status","plugin:elasticsearch@5.0.0","info"],"pid":18437,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-11-17T07:45:05Z","tags":["status","plugin:xpack_main@5.0.0","info"],"pid":18437,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-11-17T07:45:05Z","tags":["status","plugin:monitoring@5.0.0","info"],"pid":18437,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-11-17T07:45:05Z","tags":["status","plugin:console@5.0.0","info"],"pid":18437,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-11-17T07:45:06Z","tags":["status","plugin:timelion@5.0.0","info"],"pid":18437,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-11-17T07:45:06Z","tags":["status","plugin:elasticsearch@5.0.0","info"],"pid":18437,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-11-17T07:45:06Z","tags":["listening","info"],"pid":18437,"message":"Server running at http://localhost:5601"}
{"type":"log","@timestamp":"2016-11-17T07:45:06Z","tags":["status","ui settings","info"],"pid":18437,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-11-17T07:45:07Z","tags":["license","info","xpack"],"pid":18437,"message":"Imported license information from Elasticsearch: mode: trial | status: active | expiry date: 2016-12-17T08:15:15+01:00"}
{"type":"log","@timestamp":"2016-11-17T07:45:07Z","tags":["status","plugin:monitoring@5.0.0","info"],"pid":18437,"state":"yellow","message":"Status changed from green to yellow - Waiting for Monitoring Health Check","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2016-11-17T07:45:07Z","tags":["status","plugin:xpack_main@5.0.0","info"],"pid":18437,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-11-17T07:45:11Z","tags":["status","plugin:monitoring@5.0.0","info"],"pid":18437,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Monitoring Health Check"}
This is after disabling all xpack plugins apart from monitoring. I changed owner of everything in /usr/share/kibana/optimize/bundles to kibana, and it was able to get past where it was stuck. I have no idea why kibana wouldn't have access to files created by its own installation.
Checked netstat -l again, and the correct port was being listened to now (as indicated in the logs).
Then I tried to enter Kibana and got a big red banner saying "Kibana did not load properly. Check the server output for more information."
This is my kibana access attempt from the log:
{"type":"response","@timestamp":"2016-11-17T07:48:52Z","tags":,"pid":18437,"method":"get","statusCode":200,"req":{"url":"/","method":"get","headers":{"connection":"upgrade","host":"my.domain.address","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8","accept-encoding":"gzip, deflate, sdch","accept-language":"sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":835,"contentLength":9},"message":"GET / 200 835ms - 9.0B"}
{"type":"response","@timestamp":"2016-11-17T07:48:54Z","tags":,"pid":18437,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"connection":"upgrade","host":"my.domain.address","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8","referer":"http://my.domain.address/","accept-encoding":"gzip, deflate, sdch","accept-language":"sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://my.domain.address/"},"res":{"statusCode":200,"responseTime":698,"contentLength":9},"message":"GET /app/kibana 200 698ms - 9.0B"}
{"type":"response","@timestamp":"2016-11-17T07:48:55Z","tags":,"pid":18437,"method":"get","statusCode":304,"req":{"url":"/bundles/commons.style.css?v=14438","method":"get","headers":{"connection":"upgrade","host":"my.domain.address","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36","accept":"text/css,/;q=0.1","referer":"http://my.domain.address/app/kibana","accept-encoding":"gzip, deflate, sdch","accept-language":"sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4","if-none-match":""fbbaabcdab6b8ead6e006dcf04f091eda793ca7e-gzip"","if-modified-since":"Tue, 15 Nov 2016 15:39:44 GMT"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://my.domain.address/app/kibana"},"res":{"statusCode":304,"responseTime":98,"contentLength":9},"message":"GET /bundles/commons.style.css?v=14438 304 98ms - 9.0B"}
{"type":"response","@timestamp":"2016-11-17T07:48:55Z","tags":,"pid":18437,"method":"get","statusCode":304,"req":{"url":"/bundles/kibana.style.css?v=14438","method":"get","headers":{"connection":"upgrade","host":"my.domain.address","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36","accept":"text/css,/;q=0.1","referer":"http://my.domain.address/app/kibana","accept-encoding":"gzip, deflate, sdch","accept-language":"sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4","if-none-match":""34e485f7eb3dcd2849a86362397e24e908d34a3b-gzip"","if-modified-since":"Tue, 15 Nov 2016 15:39:44 GMT"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://my.domain.address/app/kibana"},"res":{"statusCode":304,"responseTime":44,"contentLength":9},"message":"GET /bundles/kibana.style.css?v=14438 304 44ms - 9.0B"}
I tried accessing Kibana again 5 minutes later, and was able to enter it. Monitoring was available.
Edit: By the way, how is your logs looking so much better than mine? I turned on logging in kibana.yml and wrote to file.
Interesting, I wonder if there is an issue with the Ubuntu packages and disabling an X-Pack plugin. I'll try it out.
I was using the logging.dest: stdout setting, and copied the log text from my console when I had Kibana running in a non-daemon process. When logging goes to a file, the log lines have more metadata and is JSON to help you feed the logs into a service like Filebeat or Logstash.
Looks like we have an open issue regarding permissions errors after disabling a plugin: https://github.com/elastic/kibana/issues/8818
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.