Docker error during 3-node cluster running

Hi everyone, I am trying to run a 3-node cluster on my local machine. Although it doesn't happen with every run but at times I am getting the "Error response from daemon: i/o timeout" error in one of the nodes randomly and the service shuts down. I have looked in other online discussion forums for possible solutions but still couldn't find a solid resolve for this. Can anyone suggest me any possible ways to harden my deployment?

This is a docker-level error, I believe coming from moby/request.go at a98bd75256122a3e386059e1a6336ee6b82aed62 · moby/moby · GitHub

I'd check for connection issues between your docker cli (looks like probably docker compose) and your docker server. An i/o timeout could mean the docker server isn't responding or is responding too slowly.

Folks on might also have some ideas of how to troubleshoot.

1 Like

Hi @matschaffer, Yes, I was able to solve the issue by setting the vm.max_map_count parameter for the system. This post elasticsearch - How to increase vm.max_map_count? - Stack Overflow helped me.

1 Like

Interesting! Usually I see the max_map_count issue surface in the ES output rather than the docker response. For example:

$ docker run --rm -it
{"@timestamp":"2022-10-19T01:22:43.426Z", "log.level": "INFO", "message":"bound or publishing to a non-loopback address, enforcing bootstrap checks", "ecs.version": "1.2.0","":"ES_ECS","event.dataset":"elasticsearch.server","":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","":"ceb02e219d5e","":"docker-cluster"}
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log
{"@timestamp":"2022-10-19T01:22:43.433Z", "log.level": "INFO", "message":"stopping ...", "ecs.version": "1.2.0","":"ES_ECS","event.dataset":"elasticsearch.server","":"Thread-0","log.logger":"org.elasticsearch.node.Node","":"ceb02e219d5e","":"docker-cluster"}
{"@timestamp":"2022-10-19T01:22:43.472Z", "log.level": "INFO", "message":"stopped", "ecs.version": "1.2.0","":"ES_ECS","event.dataset":"elasticsearch.server","":"Thread-0","log.logger":"org.elasticsearch.node.Node","":"ceb02e219d5e","":"docker-cluster"}
{"@timestamp":"2022-10-19T01:22:43.473Z", "log.level": "INFO", "message":"closing ...", "ecs.version": "1.2.0","":"ES_ECS","event.dataset":"elasticsearch.server","":"Thread-0","log.logger":"org.elasticsearch.node.Node","":"ceb02e219d5e","":"docker-cluster"}
{"@timestamp":"2022-10-19T01:22:43.481Z", "log.level": "INFO", "message":"closed", "ecs.version": "1.2.0","":"ES_ECS","event.dataset":"elasticsearch.server","":"Thread-0","log.logger":"org.elasticsearch.node.Node","":"ceb02e219d5e","":"docker-cluster"}
{"@timestamp":"2022-10-19T01:22:43.483Z", "log.level": "INFO", "message":"Native controller process has stopped - no new native processes can be started", "ecs.version": "1.2.0","":"ES_ECS","event.dataset":"elasticsearch.server","":"ml-cpp-log-tail-thread","log.logger":"","":"ceb02e219d5e","":"docker-cluster"}

ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.

Glad you found it though!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.