Error running Elastic Maps behind proxy

[m.hanna]

I am trying to run an EMS server on a disconnected network behind an Nginx reverse-proxy in a docker swarm.
I am able to start EMS up and it looks to connect to elasticsearch without the basePath setting. As soon as I add BASEPATH to the environment EMS fails to start.
I tried to change the logging level to debug to see why it doesn't start, but I am not getting anything too obvious.

Here is the EMS stanza from my docker-compose.yml:

  ems:
    image: ems:8.11.1
    hostname: ems
    environment:
#      - BASEPATH=/map
      - ELASTICSEARCH_HOST=https://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=ems
      - ELASTICSEARCH_PASSWORD=XXXXXXXXXXXXXXX
      - ELASTICSEARCH_SSL_CERTIFICATE=/usr/src/app/server/config/certs/elasticsearch/elasticsearch.crt
      - ELASTICSEARCH_SSL_KEY=/usr/src/app/server/config/certs/elasticsearch/elasticsearch.key
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/usr/src/app/server/config/certs/ca/ca/crt
    volumes:
      - './config/ems/certs/:/usr/src/app/server/config/certs/'
    ports:
      - 8080:8080
    networks: ['net']    

With BASEPATH commented out, I can successfully connect to EMS from the elasticsearch container:

# docker exec -it elasticsearch curl -k http://ems:8080/status
{"version":"8.11.1","overall":{"level":0,"state":"available","title":"All services are available"},"services":{"License":{"level":0,"state":"available","title":"Success", ...

When I uncomment BASEPATH and restart EMS, it never enters a running state and elasticsearch cannot resolve the ems hostname. The only change I see in the logs is that the /status endpoint can no longer be resolved and gives me a status code of 404 instead of 200.

Here is the reverse-proxy snippet for completeness:

  location: /map {
    set $upstream http://ems:8080;
    proxy_pass $upstream;
    proxy_redirect off;
    proxy_set_header HOST $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $scheme;
    proxy_set_header X-Forwarded-Host $host/map;
    proxy_pass_request_headers on;
    proxy_set_header Authorization $http_authorization;
    proxy_pass_header Authorization;
  }

[m.hanna]

I seem to have gotten it somewhat working.
I removed the BASEPATH environment and modified the reverse proxy.

My current proxy settings are:

  location: /map/ {
      proxy_pass http://ems:8080/;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
      proxy_set_header Host $host;
}

This allows kibana to resolve the EMS server with map.emsUrl: https://<host>/map set.
But this has a downside in that I cannot use variables in the proxy_pass setting. If the EMS server is down, then my reverse proxy will not start because it will try to verify that https://ems:8080 is accessible, so none of the other services will be reachable.
Also, I cannot see any of the layers from the EMS UI. I am not convinced that is related to this issue; I might need to play with the reverse proxy settings some more.

Also, I wonder why basePath didn't seem to work as intended.

[jsanz (Jorge Sanz)]

hi @m.hanna thanks for reporting the issue, we'll take a look a this next week and get back to you with any findings. Thanks also for sharing your attempts as well, everything helps.

[jsanz (Jorge Sanz)]

Can't check deeper now, but I did a quick test, and the basePath setting is working on my local environment (8.11) both as being passed as an environment variable and in the configuration file.

Maybe it is better to contact your Elastic support representative to get a discussion where logs or other private details are better handled?

Best regards.

[m.hanna]

Thanks for looking into it.
It looks to me that the main issue with setting basePath is the that /status endpoint can no longer be resolved (I get a 404 instead of the expected 200). So then the healthcheck in the docker container fails.
For running EMS with a regular docker compose, this just means that EMS will show as unhealthy, but will still start.
In a docker swarm, however, EMS fails to start at all when the container is unhealthy. I will play around with adding my own healthcheck in the compose file and see if that resolves the issue.

[jsanz (Jorge Sanz)]

@m.hanna

In a docker swarm, however, EMS fails to start at all when the container is unhealthy. I will play around with adding my own healthcheck in the compose file and see if that resolves the issue.

This is unfortunate

We'll take a look to see if there's a fix we can add to ensure the healthcheck honors the basePath setting.

Please, keep us informed if you can override it.

[m.hanna]

Definitely looks like the heathcheck settings in the container.
As a quick test, I added BASEPATH=/map back to the environment, changed the reverse proxy back to the original settings (first post), and then added

  healthcheck:
    disable: true

to my compose file and EMS started up fine.
Not only does Kibana see the map server using the correct URL, but the EMS UI now also works as expected.

[jsanz (Jorge Sanz)]

Glad that worked

We'll fix this issue and report back here when we know in which version exactly this is released.

Feel free to ping us here anytime.
Best