Is Docker Compose supported for a single node Elasticsearch setup?

Elastic Stack Elasticsearch
1

I'm new to Elasticstack and trying to set up a minimal development installation.

I followed the instructions in Install Kibana With Docker and got what I needed. However, it seems silly to run multiple docker run commands every time I want to bring the server up or down since that is what Docker compose is for.

I just spent a few days trying to create a minimal Elastic stack using Docker compose and was unable to do so. I went down a lot of blind alleys, but the problem ultimately came down to the fact that the automatic setup where the security information is printed to the screen does not work when running compose.

This is described in Issue 88272. This issue is closed and looks like it won't be fixed and recommends that people bring up further questions on the Elasticsearch forums.

What are my options if nevertheless I want to use Docker compose for a simple dev installation? Is there some way I can set up security with raw docker run commands and then transition over to using a Docker compose file? (I've been trying to do this, and I think the answer is "no".) Do I have to manually configure the security? (I tried unsuccessfully to do that. I guess I could eventually figure it out, but that's a lot of work.)

There are instructions on how to Start a multi-node cluster with Docker Compose. I started trying to adapt it for a single node, but the Docker compose file has this big unwieldy bash script written into it, which I started to modify and then gave up on. Again, I could eventually figure it out, but that's difficult, and having to deal with an unwieldily bash script in order to do an install defeats the purpose of having Docker in the first place. :grinning:

Is there an easy way to make a single-node docker-compose.yml setup that I'm overlooking or is that not supported?

1 Like
2

HI @wpm Welcome to the community!

I am sure you were close... here is mine it is just a pared down from the one you referenced.

This plus the .env file and the instructions worked fine for me

version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local
3

Thanks for the example. I'll probably just stick with a short shell script for my service coordination instead of docker compose to reduce the complexity I have to deal with.

But reading through your example brought back a question I had when I was trying to manually set up security. As far as I can tell, the script in the compose file above is just running the steps necessary to set up basic security for the Elastic stack. The compose script and the linked official documentation seem to be doing the same thing, except the script has this additional step where you change the file permissions on the certificate files which the documentation never mentions.

Say I did want to set up basic security manually, do I have to do some file permission work in addition to the steps outlined in the documentation?

I ask because I did try to follow the manual setup steps and at one point got stopped by a mysterious error that I worked around by setting global read write permissions for the certificate files. Or at least I think that's how I got around it. I was deep in the weeds at that point and just trying a bunch of stuff so I can't be sure what worked, but I think file permissions was it.

4

For Elastic 8.0 and greater security is automatically turned on. Is there some way I can turn it off? There are a few YouTube tutorials that do exactly what I want to with Docker compose, but they were all made with pre-8.0 versions of Elastic.

I'm not building production environments. I'm doing lots of quick and dirty proofs of concept for which having all the Docker bells-and-whistles is helpful.

I'm willing to make the effort to manually set up security when it comes time to go into production, but for right now it's less important that I have security and more important that I can show my colleagues a ten-line Docker compose file and say, "See how easy that is?"

5

Here is my docker compose without security yup I use it for a lot of quick stuff...

I use the ${TAG} syntax so I can chose a version without having to code in the docker compose file

then use like

$ TAG=8.4.0 docker-compose up

Note 'xpack.security.enabled=false', 'xpack.security.enrollment.enabled=false'

---
version: '3'
services:
  elasticsearch:
    container_name: es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
    # 8.x
    environment: ['ES_JAVA_OPTS=-Xms2g -Xmx2g','bootstrap.memory_lock=true','discovery.type=single-node','xpack.security.enabled=false', 'xpack.security.enrollment.enabled=false']
    
    # 7.17.X
    # environment: ['ES_JAVA_OPTS=-Xms2g -Xmx2g','bootstrap.memory_lock=true','discovery.type=single-node']

    ports:
      - 9200:9200
    networks:
      - elastic
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536

  kibana:
    image: docker.elastic.co/kibana/kibana:${TAG}
    container_name: kib01
    environment:
      XPACK_APM_SERVICEMAPENABLED: "true"
      XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: d1a66dfd-c4d3-4a0a-8290-2abcb83ab3aa
      LOGGING_ROOT_LEVEL: error


    ports:
      - 5601:5601
    networks:
      - elastic

networks:
  elastic:
3 Likes
6

Thanks. That no security setup is exactly what I was looking for.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.