Hi,
When bringing up a docker container for ES, I am seeing this error:
ubuntu@ip-<>:~/ragflow/docker$ sudo docker inspect --format "{{json .State.Health }}" ragflow-es-01 | jq
{
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2024-06-23T03:59:52.150793945Z",
"End": "2024-06-23T03:59:59.146252217Z",
"ExitCode": 7,
"Output": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\ncurl: (7) Failed to connect to localhost port 9200: Connection refused\n"
},...
]
}
Some System info:
EC2 Instance: ubuntu-noble-24.04-amd64-server
vm.max_map_count = 262144
Docker info:
STACK_VERSION=8.11.3
ES_PORT=1200
CLUSTER_NAME=rag_flow
TIMEZONE='Asia/Shanghai'
MEM_LIMIT=8073741824
es01:
container_name: ragflow-es-01
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- esdata01:/usr/share/elasticsearch/data
ports:
- ${ES_PORT}:9200
environment:
- node.name=es01
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=false
- xpack.security.enabled=false
- cluster.max_shards_per_node=4096
- TZ=${TIMEZONE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test: ["CMD-SHELL", "curl http://localhost:9200"]
interval: 10s
timeout: 10s
retries: 120
networks:
- ragflow
restart: always
I have already modified the security group settings to allow All TCP, SSH, HTTP, HTTPS (both inbound and outbound) [NOTE- ONLY TEMPORARILY FOR TRIAGE]. However the port 9200 isn't open.
ubuntu@ip-~/ragflow/docker$ nmap -p- localhost
Starting Nmap 7.94SVN ( https://nmap.org ) at 2024-06-22 07:12 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.029s latency).
Not shown: 65528 closed tcp ports (conn-refused)
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
1200/tcp open scol
5455/tcp open apc-5455
6379/tcp open redis
9000/tcp open cslistener
9001/tcp open tor-orport
Nmap done: 1 IP address (1 host up) scanned in 35.03 seconds
These are the docker ps that are running:
ubuntu@ip-<>:~/ragflow/docker$ sudo docker compose ps
nmap WARN[0001] The "MACOS" variable is not set. Defaulting to a blank string.
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
ragflow-es-01 docker.elastic.co/elasticsearch/elasticsearch:8.11.3 "/bin/tini -- /usr/l…" es01 55 minutes ago Up About a minute (health: starting) 9300/tcp, 0.0.0.0:1200->9200/tcp, :::1200->9200/tcp
ragflow-minio quay.io/minio/minio:RELEASE.2023-12-20T01-00-02Z "/usr/bin/docker-ent…" minio 22 hours ago Up About a minute 0.0.0.0:9000-9001->9000-9001/tcp, :::9000-9001->9000-9001/tcp
ragflow-mysql mysql:5.7.18 "docker-entrypoint.s…" mysql 22 hours ago Up About a minute (healthy) 0.0.0.0:5455->3306/tcp, :::5455->3306/tcp
ragflow-redis redis:7.2.4 "docker-entrypoint.s…" redis 22 hours ago Up About a minute 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp
docker compose logs just state this:
ragflow-es-01 | Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
ragflow-es-01 |
ragflow-es-01 | ERROR: Elasticsearch exited unexpectedly, with exit code 137
This is the elastisearch.yml from within the container
root@aefbd24acf01:/usr/share/elasticsearch/config# cat elasticsearch.yml
cluster.name: "docker-cluster"
network.host: 0.0.0.0
And these are the logs from within the docker container itself:
root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.14
[2024-06-23T04:16:33.223+0000][143][info][gc,init] CardTable entry size: 512
[2024-06-23T04:16:33.345+0000][143][info][gc ] Using G1
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Version: 21.0.1+12-29 (release)
[2024-06-23T04:16:33.378+0000][143][info][gc,init] CPUs: 1 total, 1 available
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Memory: 957M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Large Page Support: Disabled
[2024-06-23T04:16:33.378+0000][143][info][gc,init] NUMA Support: Disabled
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Compressed Oops: Enabled (32-bit)
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Heap Region Size: 1M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Heap Min Capacity: 8M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Heap Initial Capacity: 16M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Heap Max Capacity: 240M
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Pre-touch: Disabled
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Parallel Workers: 1
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Concurrent Workers: 1
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Concurrent Refinement Workers: 1
[2024-06-23T04:16:33.378+0000][143][info][gc,init] Periodic GC: Disabled
[2024-06-23T04:16:34.461+0000][143][info][gc,metaspace] CDS archive(s) mapped at: [0x0000762653000000-0x0000762653ca5000-0x0000762653ca5000), size 13258752, SharedBaseAddress: 0x0000762653000000, ArchiveRelocationMode: 1.
[2024-06-23T04:16:34.461+0000][143][info][gc,metaspace] Compressed class space mapped at: 0x0000762654000000-0x0000762694000000, reserved size: 1073741824
[2024-06-23T04:16:34.461+0000][143][info][gc,metaspace] Narrow klass base: 0x0000762653000000, Narrow klass shift: 0, Narrow klass range: 0x100000000
[2024-06-23T04:16:36.370+0000][143][info][safepoint ] Safepoint "Cleanup", Time since last: 1037759845 ns, Reaching safepoint: 75738001 ns, Cleanup: 21623 ns, At safepoint: 10956 ns, Total: 75770580 ns
[2024-06-23T04:16:37.571+0000][143][info][safepoint ] Safepoint "Cleanup", Time since last: 1200769188 ns, Reaching safepoint: 148555 ns, Cleanup: 23065 ns, At safepoint: 6308 ns, Total: 177928 ns
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit] Heap
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit] garbage-first heap total 18432K, used 2077K [0x00000000f1000000, 0x0000000100000000)
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit] region size 1024K, 1 young (1024K), 0 survivors (0K)
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit] Metaspace used 200K, committed 448K, reserved 1114112K
[2024-06-23T04:16:37.810+0000][143][info][gc,heap,exit] class space used 5K, committed 128K, reserved 1048576K
root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.13
[2024-06-23T04:14:12.909+0000][189][info][gc,init] CardTable entry size: 512
[2024-06-23T04:14:12.945+0000][189][info][gc ] Using G1
root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.12
[2024-06-23T04:14:08.125+0000][165][info][gc,init] CardTable entry size: 512
[2024-06-23T04:14:08.164+0000][165][info][gc ] Using G1
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Version: 21.0.1+12-29 (release)
[2024-06-23T04:14:08.206+0000][165][info][gc,init] CPUs: 1 total, 1 available
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Memory: 957M
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Large Page Support: Disabled
[2024-06-23T04:14:08.206+0000][165][info][gc,init] NUMA Support: Disabled
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Compressed Oops: Enabled (32-bit)
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Heap Region Size: 1M
[2024-06-23T04:14:08.206+0000][165][info][gc,init] Heap Min Capacity: 382M
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Heap Initial Capacity: 382M
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Heap Max Capacity: 382M
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Pre-touch: Disabled
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Parallel Workers: 1
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Concurrent Workers: 1
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Concurrent Refinement Workers: 1
[2024-06-23T04:14:08.207+0000][165][info][gc,init] Periodic GC: Disabled
[2024-06-23T04:14:08.525+0000][165][info][gc,metaspace] CDS archive(s) mapped at: [0x000070907f000000-0x000070907fca5000-0x000070907fca5000), size 13258752, SharedBaseAddress: 0x000070907f000000, ArchiveRelocationMode: 1.
[2024-06-23T04:14:08.525+0000][165][info][gc,metaspace] Compressed class space mapped at: 0x0000709080000000-0x00007090c0000000, reserved size: 1073741824
[2024-06-23T04:14:08.525+0000][165][info][gc,metaspace] Narrow klass base: 0x000070907f000000, Narrow klass shift: 0, Narrow klass range: 0x100000000
[2024-06-23T04:14:10.480+0000][165][info][safepoint ] Safepoint "Cleanup", Time since last: 1118812583 ns, Reaching safepoint: 79697048 ns, Cleanup: 11676 ns, At safepoint: 11010 ns, Total: 79719734 ns
[2024-06-23T04:14:11.640+0000][165][info][safepoint ] Safepoint "Cleanup", Time since last: 1080013631 ns, Reaching safepoint: 79996268 ns, Cleanup: 14267 ns, At safepoint: 10853 ns, Total: 80021388 ns
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit] Heap
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit] garbage-first heap total 391168K, used 3101K [0x00000000e8200000, 0x0000000100000000)
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit] region size 1024K, 2 young (2048K), 0 survivors (0K)
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit] Metaspace used 190K, committed 384K, reserved 1114112K
[2024-06-23T04:14:12.000+0000][165][info][gc,heap,exit] class space used 5K, committed 128K, reserved 1048576K
root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.11
[2024-06-23T04:14:04.772+0000][146][info][gc,init] CardTable entry size: 512
[2024-06-23T04:14:04.932+0000][146][info][gc ] Using G1
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Version: 21.0.1+12-29 (release)
[2024-06-23T04:14:05.011+0000][146][info][gc,init] CPUs: 1 total, 1 available
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Memory: 957M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Large Page Support: Disabled
[2024-06-23T04:14:05.011+0000][146][info][gc,init] NUMA Support: Disabled
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Compressed Oops: Enabled (32-bit)
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Heap Region Size: 1M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Heap Min Capacity: 8M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Heap Initial Capacity: 16M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Heap Max Capacity: 240M
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Pre-touch: Disabled
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Parallel Workers: 1
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Concurrent Workers: 1
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Concurrent Refinement Workers: 1
[2024-06-23T04:14:05.011+0000][146][info][gc,init] Periodic GC: Disabled
[2024-06-23T04:14:05.492+0000][146][info][gc,metaspace] CDS archive(s) mapped at: [0x00007798f3000000-0x00007798f3ca5000-0x00007798f3ca5000), size 13258752, SharedBaseAddress: 0x00007798f3000000, ArchiveRelocationMode: 1.
[2024-06-23T04:14:05.492+0000][146][info][gc,metaspace] Compressed class space mapped at: 0x00007798f4000000-0x0000779934000000, reserved size: 1073741824
[2024-06-23T04:14:05.492+0000][146][info][gc,metaspace] Narrow klass base: 0x00007798f3000000, Narrow klass shift: 0, Narrow klass range: 0x100000000
[2024-06-23T04:14:07.049+0000][146][info][safepoint ] Safepoint "Cleanup", Time since last: 1037539805 ns, Reaching safepoint: 40996801 ns, Cleanup: 12533 ns, At safepoint: 40104178 ns, Total: 81113512 ns
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit] Heap
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit] garbage-first heap total 18432K, used 2077K [0x00000000f1000000, 0x0000000100000000)
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit] region size 1024K, 1 young (1024K), 0 survivors (0K)
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit] Metaspace used 200K, committed 448K, reserved 1114112K
[2024-06-23T04:14:07.366+0000][146][info][gc,heap,exit] class space used 5K, committed 128K, reserved 1048576K
root@88edb4308cdb:/usr/share/elasticsearch# cat /usr/share/elasticsearch/logs/gc.log.10
[2024-06-23T04:11:42.428+0000][178][info][gc,init] CardTable entry size: 512
[2024-06-23T04:11:42.787+0000][178][info][gc ] Using G1
Am I missing something blatantly obvious? I am new to dockers and EC2.
Would appreciate any help.
Thank you