Docker error during 3-node cluster running

[tifty]

Hi everyone, I am trying to run a 3-node cluster on my local machine. Although it doesn't happen with every run but at times I am getting the "Error response from daemon: i/o timeout" error in one of the nodes randomly and the service shuts down. I have looked in other online discussion forums for possible solutions but still couldn't find a solid resolve for this. Can anyone suggest me any possible ways to harden my deployment?

[matschaffer]

This is a docker-level error, I believe coming from moby/request.go at a98bd75256122a3e386059e1a6336ee6b82aed62 · moby/moby · GitHub

I'd check for connection issues between your docker cli (looks like probably docker compose) and your docker server. An i/o timeout could mean the docker server isn't responding or is responding too slowly.

Folks on https://forums.docker.com/ might also have some ideas of how to troubleshoot.

[tifty]

Hi @matschaffer, Yes, I was able to solve the issue by setting the vm.max_map_count parameter for the system. This post elasticsearch - How to increase vm.max_map_count? - Stack Overflow helped me.

[matschaffer]

Interesting! Usually I see the max_map_count issue surface in the ES output rather than the docker response. For example:

$ docker run --rm -it docker.elastic.co/elasticsearch/elasticsearch:8.4.3
[snip]
{"@timestamp":"2022-10-19T01:22:43.426Z", "log.level": "INFO", "message":"bound or publishing to a non-loopback address, enforcing bootstrap checks", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"ceb02e219d5e","elasticsearch.cluster.name":"docker-cluster"}
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log
{"@timestamp":"2022-10-19T01:22:43.433Z", "log.level": "INFO", "message":"stopping ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"ceb02e219d5e","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-10-19T01:22:43.472Z", "log.level": "INFO", "message":"stopped", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"ceb02e219d5e","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-10-19T01:22:43.473Z", "log.level": "INFO", "message":"closing ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"ceb02e219d5e","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-10-19T01:22:43.481Z", "log.level": "INFO", "message":"closed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"ceb02e219d5e","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-10-19T01:22:43.483Z", "log.level": "INFO", "message":"Native controller process has stopped - no new native processes can be started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"ml-cpp-log-tail-thread","log.logger":"org.elasticsearch.xpack.ml.process.NativeController","elasticsearch.node.name":"ceb02e219d5e","elasticsearch.cluster.name":"docker-cluster"}

ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.

Glad you found it though!