After tinkering with elasticsearch in development, I am now ready to deploy it to production on docker on my own server, so, I was reading what practices should be followed, this documentation: Install Elasticsearch with Docker | Elasticsearch Guide [8.6] | Elastic, and I came up with the following docker command(p.s: I am not allowed to use docker compose):
docker run
-t
--name es01
-p 9200:9200
-p 9300:9300
--restart=unless-stopped
--ulimit nofile=65536:65536
-e "bootstrap.memory_lock=true"
--ulimit memlock=-1:-1
-e "discovery.type=single-node"
-e "cluster.name=es-cluster01"
-e "node.name=es01-node"
--mount source=elasticsearch,target=/usr/share/elasticsearch/data
elasticsearch:elasticsearch:8.6.1
-Xms4g -Xmx4g
Now. the issue with this one is that, its incapable of identifying -Xms4g -Xmx4g, I used this way since the documentation says I should avoid using -e ES_JAVA_OPTS="-Xms1g -Xmx1g", but since the first way did not work, I am forced to use it, my new docker command looks like this:
docker run
-t
--name es01
-p 9200:9200
-p 9300:9300
--restart=unless-stopped
--ulimit nofile=65536:65536
-e "bootstrap.memory_lock=true"
--ulimit memlock=-1:-1
-e "discovery.type=single-node"
-e "cluster.name=es-cluster01"
-e ES_JAVA_OPTS="-Xms1g -Xmx1g"
-e "node.name=es01-node"
--mount source=elasticsearch,target=/usr/share/elasticsearch/data
elasticsearch:elasticsearch:8.6.1
When I use this command to attempt to deploy, I get the following error which prevents elasticsearch installation from commencing: "log.level":"ERROR", "message":"exception during geoip databases update",, it never reaches the part where it displays the password and other tokens.
I know that the issue has to do with the -e ES_JAVA_OPTS="-Xms1g -Xmx1g" part since when I tried to deploy elasticsearch without it, it deployed correctly, without any initial problems. the command I used:
docker run
-t
--name es01
-p 9200:9200
-p 9300:9300
--restart=unless-stopped
--ulimit nofile=65536:65536
-e "bootstrap.memory_lock=true"
--ulimit memlock=-1:-1
-e "discovery.type=single-node"
-e "cluster.name=es-cluster01"
-e "node.name=es01-node"
--mount source=elasticsearch,target=/usr/share/elasticsearch/data
elasticsearch:elasticsearch:8.6.1
I monitored its performance before the actual deployment, and when I checked the next day I found out that the container has exited, I checked the logs, but there is no indication for why the container exited, all I have as a form of acknowledgment that the container has exited is these logs:
{"@timestamp":"2023-02-16T14:41:09.222Z", "log.level": "INFO", "message":"stopping ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.node.Node","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:09.275Z", "log.level": "INFO", "message":"shutting down watcher thread", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[file-settings-watcher]","log.logger":"org.elasticsearch.reservedstate.service.FileSettingsService","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:09.370Z", "log.level": "INFO", "message":"watcher service stopped", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.reservedstate.service.FileSettingsService","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:09.433Z", "log.level": "INFO", "message":"stopping watch service, reason [shutdown initiated]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.xpack.watcher.WatcherService","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:09.442Z", "log.level": "INFO", "message":"watcher has stopped and shutdown", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01-node][generic][T#5]","log.logger":"org.elasticsearch.xpack.watcher.WatcherLifeCycleService","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:09.918Z", "log.level": "INFO", "message":"[controller/85] [Main.cc@176] ML controller exiting", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"ml-cpp-log-tail-thread","log.logger":"org.elasticsearch.xpack.ml.process.logging.CppLogMessageHandler","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:09.924Z", "log.level": "INFO", "message":"Native controller process has stopped - no new native processes can be started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"ml-cpp-log-tail-thread","log.logger":"org.elasticsearch.xpack.ml.process.NativeController","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:10.621Z", "log.level": "INFO", "message":"stopped", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.node.Node","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:10.622Z", "log.level": "INFO", "message":"closing ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.node.Node","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:10.849Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-11030860248288105929/geoip-databases/AMDRWRZ5Rr6UcWV7ptm8Ow/GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:10.852Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-11030860248288105929/geoip-databases/AMDRWRZ5Rr6UcWV7ptm8Ow/GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:10.853Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-11030860248288105929/geoip-databases/AMDRWRZ5Rr6UcWV7ptm8Ow/GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
{"@timestamp":"2023-02-16T14:41:10.893Z", "log.level": "INFO", "message":"closed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-0","log.logger":"org.elasticsearch.node.Node","elasticsearch.cluster.uuid":"_QNP62grT5eXUpmHugOkQg","elasticsearch.node.id":"AMDRWRZ5Rr6UcWV7ptm8Ow","elasticsearch.node.name":"es01-node","elasticsearch.cluster.name":"es-cluster01"}
I beautified the logs a bit:
| index | timestamp | log.level | message | ecs.version | service.name | event.dataset | process.thread.name | log.logger | elasticsearch.cluster.uuid | elasticsearch.node.id | elasticsearch.node.name | elasticsearch.cluster.name |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2023-02-16T14:41:09.222Z | INFO | stopping ... | 1.2.0 | ES_ECS | elasticsearch.server | Thread-0 | org.elasticsearch.node.Node | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 2 | 2023-02-16T14:41:09.275Z | INFO | shutting down watcher thread | 1.2.0 | ES_ECS | elasticsearch.server | elasticsearch[file-settings-watcher] | org.elasticsearch.reservedstate.service.FileSettingsService | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 3 | 2023-02-16T14:41:09.370Z | INFO | watcher service stopped | 1.2.0 | ES_ECS | elasticsearch.server | Thread-0 | org.elasticsearch.reservedstate.service.FileSettingsService | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 4 | 2023-02-16T14:41:09.433Z | INFO | stopping watch service, reason [shutdown initiated] | 1.2.0 | ES_ECS | elasticsearch.server | Thread-0 | org.elasticsearch.xpack.watcher.WatcherService | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 5 | 2023-02-16T14:41:09.442Z | INFO | watcher has stopped and shutdown | 1.2.0 | ES_ECS | elasticsearch.server | elasticsearch[es01-node][generic][T#5] | org.elasticsearch.xpack.watcher.WatcherLifeCycleService | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 6 | 2023-02-16T14:41:09.918Z | INFO | [controller/85] [Main.cc@176] ML controller exiting | 1.2.0 | ES_ECS | elasticsearch.server | ml-cpp-log-tail-thread | org.elasticsearch.xpack.ml.process.logging.CppLogMessageHandler | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 7 | 2023-02-16T14:41:09.924Z | INFO | Native controller process has stopped - no new native processes can be started | 1.2.0 | ES_ECS | elasticsearch.server | ml-cpp-log-tail-thread | org.elasticsearch.xpack.ml.process.NativeController | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 8 | 2023-02-16T14:41:10.621Z | INFO | stopped | 1.2.0 | ES_ECS | elasticsearch.server | Thread-0 | org.elasticsearch.node.Node | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 9 | 2023-02-16T14:41:10.622Z | INFO | closing ... | 1.2.0 | ES_ECS | elasticsearch.server | Thread-0 | org.elasticsearch.node.Node | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 10 | 2023-02-16T14:41:10.849Z | INFO | evicted [0] entries from cache after reloading database [/tmp/elasticsearch-11030860248288105929/g... | 1.2.0 | ES_ECS | elasticsearch.server | Thread-0 | org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 11 | 2023-02-16T14:41:10.852Z | INFO | evicted [0] entries from cache after reloading database [/tmp/elasticsearch-11030860248288105929/geoip-databases/AMDRWRZ5Rr6UcWV7ptm8Ow/GeoLite2-ASN.mmdb] | 1.2.0 | ES_ECS | elasticsearch.server | Thread-0 | org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 12 | 2023-02-16T14:41:10.853Z | INFO | evicted [0] entries from cache after reloading database [/tmp/elasticsearch-11030860248288105929/geoip-databases/AMDRWRZ5Rr6UcWV7ptm8Ow/GeoLite2-City.mmdb] | 1.2.0 | ES_ECS | elasticsearch.server | Thread-0 | org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
| 13 | 2023-02-16T14:41:10.893Z | INFO | closed | 1.2.0 | ES_ECS | elasticsearch.server | Thread-0 | org.elasticsearch.node.Node | _QNP62grT5eXUpmHugOkQg | AMDRWRZ5Rr6UcWV7ptm8Ow | es01-node | es-cluster01 |
I tested the same container once again, perhaps it was a server issue, but no, once again, it exited, with the same issue.