Hi, I am using ELK 8.3 , 3 servers. 2 storage servers and one for cluster.
Initially I've got one server, when I've run out of free space I've added another one server for storage and third, because you need 3 servers (elk, elk2, elk3) for cluster. Now I'am running out of space on both server and I can't understand what's happening, please help.
I see that all shards are on 2nd server
GET /_cat/shards?v&bytes=gb
net-2024.11.12 0 p STARTED 162287087 13 13 192.168.18.221 elk2
net-2024.11.12 0 r UNASSIGNED
net-2024.05.21 0 p STARTED 168674245 14 14 192.168.18.221 elk2
net-2024.05.21 0 r UNASSIGNED
net-2024.03.20 0 p STARTED 157339574 13 13 192.168.18.221 elk2
net-2024.05.04 0 p STARTED 165000099 14 14 192.168.18.221 elk2
net-2024.09.24 0 p STARTED 155278336 13 13 192.168.18.221 elk2
net-2024.09.24 0 r UNASSIGNED
.slo-observability.summary-v2 0 p STARTED 0 0 0 192.168.18.221 elk2
But there is no space on both servers
GET /_cat/nodes?v&h=name,ip,disk.used_percent,disk.total,disk.used
name ip disk.used_percent disk.total disk.used
elk3 192.168.18.220 15.37 36.1gb 5.5gb
elk2 192.168.18.221 92.35 8.1tb 7.4tb
elk 192.168.18.222 96.14 3.4tb 3.3tb
all segments are one the 2nd server
GET /_cat/segments?v&bytes=gb
net-2024.05.10 0 p 192.168.18.221 _0 0 38757 0 0 0 true true 9.8.0 true
net-2024.05.10 0 p 192.168.18.221 _1 1 28487 0 0 0 true true 9.8.0 true
net-2024.05.10 0 p 192.168.18.221 _2 2 41548 0 0 0 true true 9.8.0 true
net-2024.05.10 0 p 192.168.18.221 _3 3 53654 0 0 0 true true 9.8.0 true
net-2024.05.10 0 p 192.168.18.221 _4 4 44436 0 0 0 true true 9.8.0 true
There is two question:
- How find what is filling up space on first server an free it up (it's not a logs, I've checked)
- How to make Elastic to use both server to store data ? I just want to split data e.g. 1,3,5 shards on 1st server and 2,4,6 shards on 2nd server