Nofile -> limit for open file under docker container for elasticsearch user

INS · March 15, 2022, 4:39pm

Hi I read this post https://discuss.elastic.co/t/docs-nofile-or-nproc/142049/2 regarding construct for nofile vs nproc.
Let me describe my case
I've provisioned by docker compose service for elasticsearch:

ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65535
        hard: 65535
    deploy:

at least I saw in docker container that I have

open files (-n) 1048576


elasticsearch@441ecdf403aa:~$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1029415
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

from elaticsearch point of view it's also doesn't look as expected:

"wOMZvpx7QkW1IoDB9-hv5A" : {
      "timestamp" : 1647361899514,
      "name" : "es_coordination_1",
      "transport_address" : "10.0.9.187:9300",
      "host" : "10.0.9.187",
      "ip" : "10.0.9.187:9300",
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "remote_cluster_client",
        "transform"
      ],
      "attributes" : {
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "process" : {
        "timestamp" : 1647361899516,
        "open_file_descriptors" : 1771,
        "max_file_descriptors" : 1048576,

on the system I've also:
config in /etc/security/limits.conf

elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

Have You ever met such kind of case?

INS · March 15, 2022, 5:49pm

doesn't swarm support ulimits: ??? yet

INS · March 15, 2022, 6:51pm

Already found the solution

github.com/docker/cli

Add ulimits support to docker service and docker stack deploy (carry 2660)

docker:master ← thaJeztah:carry_2660_ulimits

opened 12:43PM - 10 Sep 20 UTC

thaJeztah

+252 -23

carries https://github.com/docker/cli/pull/2660 closes https://github.com/docke…r/cli/pull/2660 **- What I did** 1. ~Bump docker/docker to my own fork to have API changes regarding ulimits support on service endpoints ;~ 2. Add `--ulimit` to `docker service create` ; 3. Add `--ulimit-add` and `--ulimit-rm` options to `docker service update` ; 4. Add `Ulimits` to `docker service inspect --pretty` ; 5. Support ulimits in docker-compose files, to make them work with `docker stack deploy` ; This is related to moby/moby#40639. **- How I did it** **- How to verify it** Given the following `docker-compose.yaml`: ```yaml version: "3" services: test: image: debian command: /bin/bash -c "ulimit -a && sleep 15" ulimits: nofile: 100 deploy: mode: replicated ``` <details> <summary><code>docker service create</code></summary> ``` $ docker service create --name t2 --ulimit=nofile=100 debian /bin/bash -c "ulimit -a && sleep 30" $ docker service logs -f t2 ... t2.1.dkm9duj5i4rq@aker | open files (-n) 100 t2.1.dkm9duj5i4rq@aker | real-time priority (-r) 0 ... ``` </details> <details> <summary><code>docker service update</code></summary> ``` $ docker service update --ulimit-rm nofile --ulimit-add rtprio=1 t2 $ docker service logs -f t2 ... t2.1.dkm9duj5i4rq@aker | open files (-n) 1048576 t2.1.dkm9duj5i4rq@aker | real-time priority (-r) 1 ... ``` </details> <details> <summary><code>docker service inspect</code></summary> ``` $ docker service update --ulimit-add nofile=100:200 t2 $ docker service inspect --pretty t2 ... Ulimits: nofile: 100:200 nproc: -1:-1 ... $ docker service inspect t2 ... "Ulimits": [ { "Name": "nproc", "Hard": -1, "Soft": -1 }, { "Name": "nofile", "Hard": 200, "Soft": 100 } ] ... ``` </details> <details> <summary><code>docker stack deploy</code></summary> ``` $ cat docker-compose.yaml version: "3" services: test: image: debian command: /bin/bash -c "ulimit -a && sleep 15" ulimits: nofile: 100 deploy: mode: replicated $ docker stack deploy --compose-file docker-compose.yaml test $ docker service logs -f test_test | grep "open files" test_test.1.jezlw4wt08rz@aker | open files (-n) 100 $ docker inspect $(docker ps --filter=name=test_test -q) ... "Ulimits": [ { "Name": "nofile", "Hard": 100, "Soft": 100 } ], ... ``` </details> **- Description for the changelog** Add `ulimits` support to `docker service create|update|inspect` and `docker stack deploy` **- A picture of a cute animal (not mandatory but encouraged)**

system · April 12, 2022, 6:51pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Ulimits for elasticsearch instances Elasticsearch docker	8	4159	April 18, 2022
[1]: max number of threads [2048] for user [elasticsearch] is too low, increase to at least [4096] Elasticsearch	1	6929	September 28, 2018
Docs: nofile or nproc? Elasticsearch	3	2021	August 26, 2018
Maximum number of open files Elasticsearch docker	1	431	January 20, 2020
Elasticsearch docker ulimit Elasticsearch	1	2344	April 12, 2018

Nofile -> limit for open file under docker container for elasticsearch user

Related topics