Docker host node Java opts

Elastic Stack Elasticsearch
1

I am in the process of going from dev to a production model of the Elastic stack with docker compose. I have a few newbie questions about java mem and heap sizes. The docker host box has a total of 24gb of ram. In my docker compose file each elastic node is set to

- "ES_JAVA_OPTS=-Xms512m -Xmx512m"

Do I need to set each node to only use a percentage of the box? Also is there any guidelines as to what to leave left over for the host machine? Should I give each node 6gb or ram and leave the host machine 6gb?

Also in the docker-compose.yml, each node has ulimts set. I am a little confused by the documentation as to what these need to be in production. Currently, they are set at mem_limit 1g. Should this value reflect the total ram in the host machine? (mem_limit: 24gb)

Full docker-compose.yml

version: '2'
services:
  elasticsearch1:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.4
    container_name: elasticsearch1
    environment:
      - cluster.name=SASD-Docker-Cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.4
    container_name: elasticsearch2
    environment:
      - cluster.name=SASD-Docker-Cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch1, elasticsearch3"
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet
  elasticsearch3:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.4
    container_name: elasticsearch3
    environment:
      - cluster.name=SASD-Docker-Cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch1, elasticsearch2"
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet    
  kibana:
    image: docker.elastic.co/kibana/kibana:5.6.4
    environment:
      SERVER_NAME: s50-kibana.company.com
      ELASTICSEARCH_URL: http://elasticsearch1:9200
      ELASTICSEARCH_USERNAME: elastic
      ELASTICSEARCH_PASSWORD: changeme
    ports:
      - "5601:5601"
    networks:
      - esnet
  logstash:
    image: docker.elastic.co/logstash/logstash:5.6.4
    environment:
      - JAVA_OPTS=-Xms1g -Xmx1g
      - XPACK_MONITORING_ENABLED=false
    volumes:
      - /home/ncssdocker/pipeline/:/usr/share/logstash/pipeline/
    command: logstash -f /usr/share/logstash/pipeline/FiresightKV.conf
    ports:
      - "5000:5000/udp"
    networks:
      - esnet
    depends_on:
      - elasticsearch1
volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local
networks:
  esnet:
2

First of all, it is not generally recommended, for a production environment, to run multiple nodes per host . This is only barely recommended with hosts that features +100GB of RAM, which is not your case.

That being said, if you only have a single host then you should run only a single node on this host and assign 50% of available RAM to JVM (this is the recommendation for initial setup). The reason for that is because great part of Elasticsearch performance relies on OS-level filesystem cache.

Regarding the mem_limit, that setting is the hard memory limit for Docker. In the example of the 5.6 docs the JVM is set to 512MB while the memory limit is 1G (50% of available memory is set to JVM). That setting was there as an example, but since can be confusing, it was removed from the 6.0 docs. In your case, you should not be using it and leave all the rest of available RAM for filesystem cache.

3

Thanks for clearing up the mem_limit. I guess I don't see the point of using docker then if each node needs its own host. I was under the impression that I could set container limits within Docker as to what each node container could use.

Just confusing in the docker installation docs from Elastic since the example they give is a docker-compose that spins up a 2 node cluster on the same host. It also doesn't give any guidelines in the production notes that warns against running multiple nodes on the same docker host.

https://www.elastic.co/guide/en/elasticsearch/reference/6.0/docker.html

Thanks

4

What I have explained to you a whole different topic regarding architecture design that is not the focus of that docker documentation. The samples there are focused solely on docker and docker compose (it would be pointless to feature a docker compose sample with just one node) and does not takes architectural details into account (otherwise the documentation would be a mess).

It's a little bit outdated (and we are fixing that) but still a valid documentation about production deployment: https://www.elastic.co/guide/en/elasticsearch/guide/current/administration.html

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.