Elasticsearch settings do not seem to be applied via Docker env variables?

[jerrac (David Reagan)]

I'm running a small test cluster via docker-compose on a desktop in my office, I've configured each node via environment variables. Not by using elasticsearch.yml.

My main node looks like:

version: '3.3'
services:
  esnode1:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
    container_name: esnode1
    environment:
      - cluster.name=reagan3-cluster
      - node.name=esnode1
      - node.master=true
      - discovery.seed_hosts=esnode1,esnode2,esnode3
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms6000m -Xmx6000m"
      - http.cors.enabled=true
      - http.cors.allow-origin="*"
      - http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
      - http.cors.allow-credentials=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - r3_cluster_esdata1:/usr/share/elasticsearch/data
      - r3_cluster_snapshots:/opt/elasticsearch/snapshots
    ports:
      - 127.0.0.1:9201:9200
    healthcheck:
      test: ["CMD", "curl","-s" ,"-f", "http://localhost:9200/_cat/health"]
    networks:
      - elknet
    restart: always

I have Apache running as a reverse proxy in front of ES so I can use LDAP auth.

I've been trying to test Mirage and Dejavu from my laptop, a simple curl call with my username and password works just fine, so the proxy is working. I can also access my Kibana instance just fine, and my cluster health is green.

But when I try to connect to the cluster using Mirage or Dejavu, Firefox's console spits out errors like this:

Mirage:

This page uses the non standard property “zoom”. Consider using calc() in the relevant property values, or using “transform” along with “transform-origin: 0 0”. localhost:3030
Source map error: Error: request failed with status 404
Resource URL: http://localhost:3030/dist/css/vendor.min.css
Source Map URL: bootstrap.min.css.map
Angular 2 is running in the development mode. Call enableProdMode() to enable the production mode. core.umd.js:201:17
Object { url: "http://username:password@internalhostname:9200", appname: "metricbeat-7.5.1-2019.12.29", username: "", password: "", host: "" }
 
Object { username: "username", password: "password", url: "http://internalhostname:9200" }
app.component.ts:336:12
setting up appbase appbase.service.ts:73:12
http://internalhostname:9200/metricbeat-7.5.1-2019.12.29/_settings appbase.service.ts:182:12
http://internalhostname:9200/metricbeat-7.5.1-2019.12.29/_mapping/ appbase.service.ts:163:14
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://internalhostname:9200/metricbeat-7.5.1-2019.12.29/_mapping/. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://internalhostname:9200/metricbeat-7.5.1-2019.12.29/_settings. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://internalhostname:9200/metricbeat-7.5.1-2019.12.29/_mapping/. (Reason: CORS request did not succeed).
Object { _body: error, status: 0, ok: false, statusText: "", headers: {…}, type: 3, url: null }
app.component.ts:446:16
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://internalhostname:9200/metricbeat-7.5.1-2019.12.29/_settings. (Reason: CORS request did not succeed).
Not able to get the version. app.component.ts:382:16

Dejavu:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://internalhostname:9200/metricbeat-7.5.1-2019.12.29. (Reason: CORS request did not succeed).
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://internalhostname:9200/metricbeat-7.5.1-2019.12.29. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://internalhostname:9200/metricbeat-7.5.1-2019.12.29. (Reason: CORS request did not succeed).

I also tried a curl request on my desktop that set the Origin header. (Found the idea here )

$ curl --head http://localhost:9201 -H 'Origin: http://foo.com'
HTTP/1.1 403 Forbidden

Judging from the docs, http.cors.allow-origin="*" should open my cluster up to any origin, correct?

That, combined with the "node attributes" error I mention here, makes me wonder if my settings are just not being applied correctly.

Is that the case? Do I have to use elasticsearch.yml to get those settings applied? Any other ideas?

[jerrac (David Reagan)]

So, I've created a test cluster on my laptop with no authentication. I bind mount the elasticsearch.yml file to my elasticsearch container. And I'm still getting cors errors.

Note: I've had this mini cluster running for a couple weeks getting data pushed into it from some vagrant virtual machines.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://learnes1:9200/filebeat-7.5.1-2020.01.14. (Reason: CORS request did not succeed).

My docker-compose.yml file:

version: '3.3'
volumes:
  learnes_cluster_esdata1:
    driver: local
  learnes_cluster_esdata2:
    driver: local
  learnes_cluster_eslogs1:
    driver: local
  learnes_cluster_eslogs2:
    driver: local

services:
  learnes1:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
    container_name: learnes1
    environment:
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./config/elasticsearch/learnes1/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - learnes_cluster_esdata1:/usr/share/elasticsearch/data
      - learnes_cluster_eslogs1:/usr/share/elasticsearch/logs
    ports:
      - 9200:9200
    healthcheck:
      test: ["CMD", "curl","-s" ,"-f", "http://localhost:9200/_cat/health"]
    networks:
      - elknet
    restart: always
  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.5.1
    environment:
      #SERVER_NAME: 127.0.0.1
      ELASTICSEARCH_HOSTS: http://learnes1:9200
      SERVER_HOST: 0.0.0.0
      SERVER_PORT: 5601
      ELASTICSEARCH_USERNAME: kibana
      ELASTICSEARCH_PASSWORD: password
#      SERVER_BASEPATH: /kibana
#      SERVER_REWRITEBASEPATH: "true"
    ports:
      - 5601:5601
    networks:
      - elknet
    restart: always
    depends_on:
      - learnes1
  # logstash:
  #   container_name: learnes_logstash
  #   image: docker.elastic.co/logstash/logstash:7.5.1
  #   volumes:
  #     - ./config/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml
  #     - ./config/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
  #     - ./config/logstash/elasticsearch-template-es7x.json:/usr/share/logstash/config/elasticsearch-template-es7x.json
  #     # - /srv/logstash/data/logstash/:/srv/
  #   networks:
  #     - elknet
  #   restart: always

networks:
  elknet:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: elknet

My elasticsearch.yml file:

cluster:
  name: learnes-cluster
  # initial_master_nodes:
  #   - learnes1

node:
  name: learnes1
  master: true
path:
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs

network:
  host: _site_

discovery:
  seed_hosts:
    - learnes1
  type: single-node

http:
  cors:
    enabled: true
    allow-origin: "*"
    allow-headers: X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization,Access-Control-Allow-Origin
    allow-credentials: true


xpack:
  license:
    self_generated:
      type: basic
  ilm:
    enabled: true
  monitoring:
    enabled: false

bootstrap:
  memory_lock: true

I ran dejavu via this command:

docker run -it --rm --network elklearn_elknet -p 1358:1358 appbaseio/dejavu

And Mirage via:

docker run -it --network elklearn_elknet --rm -p 3030:3030  appbaseio/mirage

If I enter the mirage container and add curl, I can query http://learnes1:9200/_cat/health just fine, so it's not a connection error between the two containers.

So, as far as I can tell, Elasticsearch either isn't getting configured from my configuration file, or it's ignoring my cors settings.

Any ideas?

[jerrac (David Reagan)]

So, to add more confusion, I tried Dejavu in Chromium, and I received a different error.

main.48be9e90.js:1 OPTIONS http://learnes1:9200/filebeat-7.5.1-2020.01.14 net::ERR_NAME_NOT_RESOLVED

So I decided to double check the connection between my Dejavu container and my learnes1 container.

I exec'd into the Dejavu container and ran ping learnes1.

/dejavu $ ping learnes1
PING learnes1 (172.30.0.2): 56 data bytes
ping: permission denied (are you root?)

Weird permission issue aside, it does look like the container can resolve the dns name.

But seeing the ip address gave me the idea to try it instead of the hostname.

And that seems to actually work.

In both Firefox, and Chromium, and with both Dejavu and Mirage.

This all makes sense when I remember that both apps are NodeJS, so the queries to Elasticsearch are coming from my laptop. Not from within the app container. So it would be my laptop's dns lookup that would be failing.

So the issues with my laptop's elk cluster are resolved.

Unfortunately, my issues with my other cluster are still lingering. The cluster mentioned in my first post. That cluster had an dns name set in my work places internal network, so that shouldn't have been the issue. Unfortunately, that cluster is currently down due to hardware failure.

Oh well, at least I made some progress....