Nginx reverse proxy failing with csp error

[KenneyHe]

Hi,
Being a bit of a technocrat, I tried the latest 7.6.2 ES/Kibana releases and can't understand this error message after configuring nginx 1.15-alpine reverse proxy. Even after repeatedly restarting with kibana with csp.strict on/off, the error still happens... Is this a software issue? Can I workaround it? Thanks in advance...

configuration:

• latest alpine
• latest ES, Kibana
• latest nginx stable
• ubuntu 18.04 host
• latest docker 19.x
• latest docker-compose v3.3 template

steps:

# client side
$ curl https://au-001.fortressiq.com/
...
            // Since this is an unsafe inline script, this code will not run
            // in browsers that support content security policy(CSP). This is
            // intentional as we check for the existence of __kbnCspNotEnforced__ in
            // bootstrap.
            window.__kbnCspNotEnforced__ = true;

# server side docker-compose.yaml

version: '3.3'
services:
  nginx:
    image: nginx:1.15-alpine
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
      - ./data/nginx:/etc/nginx/conf.d
    networks:
      - elastic
  certbot:
    image: certbot/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.2
    container_name: kibana
    # see https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker
    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 5601:5601
    networks:
      - elastic

similary configuration as below ref:
Nginx reverse proxy with rewrite and app/kibana#/discover

using official syntax from github documentation ref:
https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker

[wylie (Wylie Conlon)]

Hi, welcome! I don't understand: what error message are you seeing?

[KenneyHe]

            // Since this is an unsafe inline script, this code will not run
            // in browsers that support content security policy(CSP). This is
            // intentional as we check for the existence of __kbnCspNotEnforced__ in
            // bootstrap.
            window.__kbnCspNotEnforced__ = true;

^
|
the error is preventing browser from displaying traditional welcome page. Hence I use curl to show this issue

[wylie (Wylie Conlon)]

That isn't an error message, it's part of the Kibana template. You will see an error message if the content security policy is not being applied.

[KenneyHe]

the nginx reverse proxy setting is below, compsoe and all the browsers support CSP. Please explain why status 200 and nothing shows up if this is not an error state, yet.

# compose
version: '3.3'
services:
  nginx:
    image: nginx:1.15-alpine
    command: "/bin/sh -c 'cat /etc/nginx/conf.d/default.conf; while :; do sleep 6h & wait $${!}; nginx -s reload; d
one & nginx -g \"daemon off;\"'"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
      - ./data/nginx:/etc/nginx/conf.d
    networks:
      - elastic
  certbot:
    image: certbot/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.2
    container_name: kibana
    # see https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/resources/
bin/kibana-docker
    environment:
        - ELASTICSEARCH_HOSTS=http://es01:9200
    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml
    networks:
      - elastic

# /etc/nginx/conf.d/default.conf is here
# nginx proxy for Elasticsearch + Kibana
server {
    listen 80;
    server_name au-001.fortressiq.com;
    location / {
        return 301 https://$host$request_uri;
    }    
    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
}
upstream docker-kibana {
    server kibana:5601;
}
server {
    listen 443 ssl;
    server_name au-001.fortressiq.com;

    set $proxy_pass_url http://docker-kibana:5601;
    location / {
      proxy_pass http://docker-kibana/app/kibana;
      proxy_set_header Host au-001.fortressiq.com;
      proxy_set_header Host $proxy_pass_url;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_http_version 1.1;
      proxy_set_header Connection "Keep-Alive";
      proxy_set_header Proxy-Connection "Keep-Alive";
      proxy_set_header Authorization "";
    #proxy_pass $proxy_pass_url/;
      proxy_redirect $proxy_pass_url/ /;
    }
    ssl_certificate /etc/letsencrypt/live/au-001.fortressiq.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/au-001.fortressiq.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

all service running:

$ docker-compose ps
      Name                    Command               State                    Ports                  
----------------------------------------------------------------------------------------------------
es01               /usr/local/bin/docker-entr ...   Up      0.0.0.0:9200->9200/tcp, 9300/tcp        
es02               /usr/local/bin/docker-entr ...   Up      9200/tcp, 9300/tcp                      
es03               /usr/local/bin/docker-entr ...   Up      9200/tcp, 9300/tcp                      
kibana             /usr/local/bin/dumb-init - ...   Up      0.0.0.0:5601->5601/tcp                  
ubuntu_certbot_1   /bin/sh -c trap exit TERM; ...   Up      443/tcp, 80/tcp                         
ubuntu_nginx_1     /bin/sh -c sudo apt-get up ...   Up      0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp

[wylie (Wylie Conlon)]

Are you using a browser that supports CSP and Javascript?

[KenneyHe]

Yes. I tried all my browsers... And, I explicitly set csp in nginx

request is successful the still Kibana claim csp issue... below is all status 200 from kibana

[wylie (Wylie Conlon)]

I can't help you with your nginx issues, but I can try to help with verifying that the Kibana CSP settings are set correctly. You still haven't shown me what the browser is showing: you've only shown server-side errors. Where is the browser error?

[KenneyHe]

ok. If you don't want to see the server-side errors, I will go to stackoverflow from now on.

[wylie (Wylie Conlon)]

Are you seeing an error message in your browser? Is your browser received a CSP header?

[KenneyHe]

I work around it with csp.strict: true in kibana. now I am seeing kibana unable to login using admin/admin default password with opendistro so not sure why kibana is behaving this way behind a proxy. Turning off kibana security is an insecure workaround, not related.

[werowe (Walker Rowe)]

I get same error. It just shows a blank page. You can only see the error if you use curl or click view source in Browser. I too am using KIbana in docker. So I gave up and took approach described below to not use nginx:

And I agree with Kenneth that engineers here need to know about nginx since practically everyone uses it that way. We do that because we need to put Kibana on a public IP.

    location /kibana {
            auth_basic "Restricted Access";
            auth_basic_user_file /etc/nginx/htpasswd.users;
            proxy_pass http://127.0.0.1:5601/app/kibana;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;

This works. Open as a shell as root and pass in SERVER_HOST which KIbana will use to expose that server on that IP so you can then open it from the public internet.

sudo -i

export SERVER_HOST="172.31.46.15"

nohup docker run --link 4a867fcd2248:elasticsearch -p 5601:5601 kibana:7.6.2&

http:://(your public ip):5601