"environment: - xpack.security.enabled=true" does not work with docker-compose

Hi, I'm trying to specify xpack.security.enabled as an environment variable for the Elasticsearch container in the docker-compose configuration of T-Pot, a honeypot framework.
However, the following statement does not work. (This setting is reflected as an environment variable in the container. See the reproduction procedure.)

    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true

Expected behavior:

$ telnet localhost 64298 (ports: - "127.0.0.1:64298:9200")
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.0 401 Unauthorized
WWW-Authenticate: Basic realm="security" charset="UTF-8"
content-type: application/json; charset=UTF-8
content-length: 381

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}Connection closed by foreign host.

Actual behavior:

$ telnet localhost 64298 (ports: - "127.0.0.1:64298:9200")
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.0 200 OK
X-elastic-product: Elasticsearch
content-type: application/json; charset=UTF-8
content-length: 546

{
  "name" : "tpotcluster-node-01",
  "cluster_name" : "tpotcluster",
  "cluster_uuid" : "vFHqJ5v5RyO1_nv_0PNPiA",
  "version" : {
    "number" : "7.15.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "83c34f456ae29d60e94d886e455e6a3409bba9ed",
    "build_date" : "2021-10-07T21:56:19.031608185Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
Connection closed by foreign host.

The reproduction procedure is as follows.

(Install T-Pot on VirtualBox with tpot.iso, and login to T-Pot virtual machine.)
PS C:\Users\tadashi.oya> ssh -l tsec -p 64295 192.168.56.102
tsec@192.168.56.102's password:
Linux cruelspike 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64
Last login: Mon Nov 29 01:35:02 2021 from 192.168.56.1
[tsec@cruelspike:~]$ sudo su -
[sudo] password for tsec:
[root@cruelspike:~]#

[root@cruelspike:~]# systemctl status tpot
...(snip)...
   Loaded: loaded (/etc/systemd/system/tpot.service; enabled; vendor preset: enabled)
...(snip)...

[root@cruelspike:~]# cat /etc/systemd/system/tpot.service
...(snip)...
# Compose T-Pot up
ExecStart=/usr/bin/docker-compose -f /opt/tpot/etc/tpot.yml up --no-color
...(snip)...

[root@cruelspike:~]# ls -l /opt/tpot/etc/tpot.yml
lrwxrwxrwx 1 root root 34 Nov 18 08:54 /opt/tpot/etc/tpot.yml -> /opt/tpot/etc/compose/standard.yml

[root@cruelspike:~]# cp /opt/tpot/etc/compose/standard.yml ~/

[root@cruelspike:~]# vi /opt/tpot/etc/compose/standard.yml
(add xpack.security.enabled and discovery.type)

[root@cruelspike:~]# diff -10 -u ~/standard.yml /opt/tpot/etc/compose/standard.yml
--- /root/standard.yml  2021-11-29 02:38:24.236000000 +0000
+++ /opt/tpot/etc/compose/standard.yml  2021-11-29 02:41:39.188000000 +0000
@@ -505,20 +505,22 @@

 #### ELK
 ## Elasticsearch service
   elasticsearch:
     container_name: elasticsearch
     restart: always
     environment:
      - bootstrap.memory_lock=true
 #     - ES_JAVA_OPTS=-Xms2048m -Xmx2048m
      - ES_TMPDIR=/tmp
+     - discovery.type=single-node
+     - xpack.security.enabled=true
     cap_add:
      - IPC_LOCK
     ulimits:
       memlock:
         soft: -1
         hard: -1
       nofile:
         soft: 65536
         hard: 65536
 #    mem_limit: 4g

[root@cruelspike:~]# systemctl stop tpot
[root@cruelspike:~]# systemctl start tpot

[root@cruelspike:~]# /opt/tpot/bin/dps.sh 15
...(snip)...
elasticsearch         Up 6 minutes (healthy)   127.0.0.1:64298->9200/tcp
...(snip)...

[root@cruelspike:~]# docker exec -it elasticsearch sh

/ $ env
HOSTNAME=2c8b01c29fee
ES_JAVA_HOME=/usr/lib/jvm/java-16-openjdk
SHLVL=1
bootstrap.memory_lock=true
HOME=/home/elasticsearch
xpack.security.enabled=true
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ES_TMPDIR=/tmp
ES_VER=7.15.1
PWD=/
discovery.type=single-node

/ $ id
uid=2000(elasticsearch) gid=2000(elasticsearch)

/ $ ps
PID   USER     TIME  COMMAND
    1 elastics  1:13 /usr/lib/jvm/java-16-openjdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkad
  259 elastics  0:00 sh
  324 elastics  0:00 ps

[root@cruelspike:~]# telnet localhost 64298 (ports: - "127.0.0.1:64298:9200")
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.0 200 OK
X-elastic-product: Elasticsearch
content-type: application/json; charset=UTF-8
content-length: 546

{
  "name" : "tpotcluster-node-01",
  "cluster_name" : "tpotcluster",
  "cluster_uuid" : "vFHqJ5v5RyO1_nv_0PNPiA",
  "version" : {
    "number" : "7.15.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "83c34f456ae29d60e94d886e455e6a3409bba9ed",
    "build_date" : "2021-10-07T21:56:19.031608185Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
Connection closed by foreign host.

Thanks.

tpot's standard.yml doesn't use the official Elasticsearch image.

I believe the image it relies on is this one: tpotce/Dockerfile at master · telekom-security/tpotce · GitHub which installs Elasticsearch from a tarball, and doesn't have the same support for configuration via docker environment variables.

You'll either need to take this up with them, or switch to using the official Elasticsearch Docker image.

Thank you for pointing that out.
I hadn't thought about the fact that Elasticsearch from a tarball doesn't support environment variables.
I have confirmed the following at hand.

$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.2-linux-x86_64.tar.gz
$ tar xvzf elasticsearch-7.15.2-linux-x86_64.tar.gz
$ cd elasticsearch-7.15.2
$ env xpack.security.enabled=true ./bin/elasticsearch

$ telnet localhost 9200
Trying ::1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.0 200 OK
X-elastic-product: Elasticsearch
Warning: 299 Elasticsearch-7.15.2-93d5a7f6192e8a1a12e154a2b81bf6fa7309da0c "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security."
content-type: application/json; charset=UTF-8
content-length: 532

{
  "name" : "elk",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "dDhohkk4RK6et4Syj2zuEw",
  "version" : {
    "number" : "7.15.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "93d5a7f6192e8a1a12e154a2b81bf6fa7309da0c",
    "build_date" : "2021-11-04T14:04:42.515624022Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
Connection closed by foreign host.

In the T-Pot configuration, it worked by mounting Elasticsearch.yml with docker-compose as shown below.

# vi /path/to/elasticsearch.yml
cluster.name: tpotcluster
node.name: "tpotcluster-node-01"
xpack.ml.enabled: false
xpack.security.enabled: true <<<HERE
#xpack.ilm.enabled: false
path:
    logs: /data/elk/log
    data: /data/elk/data
http.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
indices.query.bool.max_clause_count: 2000
cluster.initial_master_nodes:
- "tpotcluster-node-01"
discovery.zen.ping.unicast.hosts:
- localhost

# vi /opt/tpot/etc/tpot.yml
[...]
#### ELK
## Elasticsearch service
  elasticsearch:
    container_name: elasticsearch
    restart: always
    environment:
     - bootstrap.memory_lock=true
#     - ES_JAVA_OPTS=-Xms2048m -Xmx2048m
     - ES_TMPDIR=/tmp
    cap_add:
     - IPC_LOCK
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
#    mem_limit: 4g
    ports:
     - "127.0.0.1:64298:9200"
    image: "dtagdevsec/elasticsearch:2006"
    volumes:
     - /data:/data
     - /path/to/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml <<<HERE
[...]

# systemctl start tpot

And I got this as I expected:

$ telnet localhost 64298
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.0 401 Unauthorized
WWW-Authenticate: Basic realm="security" charset="UTF-8"
content-type: application/json; charset=UTF-8
content-length: 381

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}Connection closed by foreign host.

Thanks.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.