Elasticsearch is not running
There is no room in /tmp directory either
You are going to need to clean up space in /tmp and other places otherwise elasticsearch is not going to run.
all logs in directory
Which of these should I remove?
root@srvelk:/var/lib/elasticsearch/indices# ls
09byaX0KQBqbln3AiMbhnw DOHzjbOnRpagkU6D6qPfTg jLIF61xpTRieUynHu-YCMw NgSJGkuTQCCUiKJbYvO9ig QYGZ0U0_TjGgwiu5dETE5Q Wrgp26bbTlGTo6vYO3EOCQ
48n18llpS2WyrQxyvCoywA GLd0Hpj1QTKXjixqdHjBqA _keXg-3VQQewZwf07HWbaA O3LIsFtTQrqaj0spzHushA r8oGIwF-Rb-OEIHNuEPmcg W-wiCs-7ToaRr9KjOjC0Iw
74wFVcAUQ-KCz66WKjLRmw GX8XNJOVSxmU7b115yFBdA lzSS14E1RKGA9lytBFUjXQ oe2borCxQ520NlD6Ok-Y2w SbjXxyw9Q2u_WedT563dCw XTeuNnW1RECTf797ioMpoQ
A7Y5ZlG7QheXryHgK0fQpA iJ252YZ3Ty-ZWbx8MQihmA mduXWrTPSbeBzjVIGxFjlw OoAsfZ4NQGWUTPRNXt1klw sbPOoLn0QTax0nOFPAHPbQ Z6HqHHx7THeiV6lWF7wx0A
cNreQz6FQMygOhE2TnN3Uw ioJFg9NKTQCXMoayh3UaLg MoPXfnKTTbyV5YZI-QfVfw qmzZHzuwQj2DBaZWq8fplA TPlyNvgTTU64B4mAG6cn0g
D1OThSkmTx2_Gg782Sb4Qg j3Cb75X3ToGjliv06Gzwdg NAS42JsdQbqhUTnzI6dvcw QuwLdRSZTbWrw9icoF4LhA wGU1lhc3ReGtaCQMYNKdkw
du -h / | sort -rh | head -10
du: cannot access '/proc/1382/task/1382/fd/4': No such file or directory
du: cannot access '/proc/1382/task/1382/fdinfo/4': No such file or directory
du: cannot access '/proc/1382/fd/3': No such file or directory
du: cannot access '/proc/1382/fdinfo/3': No such file or directory
94G /
83G /var
82G /var/lib/elasticsearch/indices
82G /var/lib/elasticsearch
82G /var/lib
81G /var/lib/elasticsearch/indices/A7Y5ZlG7QheXryHgK0fQpA/0/index
81G /var/lib/elasticsearch/indices/A7Y5ZlG7QheXryHgK0fQpA/0
81G /var/lib/elasticsearch/indices/A7Y5ZlG7QheXryHgK0fQpA
4.9G /usr
2.7G /usr/share
Those are not logs... those are the actual elastic data.
If you remove ANY directory or files in /var/lib/elasticsearch/indices, you will lose data AND make Elasticsearch unusable. Those are not Logs they are Data.
Something is unusual... going on as Elastic is reporting ~13GB
But your du is reporting 81GB
I do not have an explanation for that perhaps there was an old Elasticsearch or there is Corrupt Data I do not have an answer.
If you want to fix this, you are going to need to find room somewhere else so that Elastic will start, then use the DELETE API to remove Indices...
Or can you expand that filesystem?
ok
I don't know where the filebeat logs are stored
No I can not. can you help me
thanks my friend
Sorry confused... not sure what filebeat has to do with this
you can look in
/var/log
and the below directories.. and perhaps clean logs out of that or in the subdirectories
How can the logs be deleted automatically after a week?
logs net flow
What logs?
Are you talking about the Elasticsearch Application Logs in /var/log/elasticsearch?
Otherwise, I can not help with the other logs.. those are up to your system admin.
hi Stephen
I install new server and after one week can not login >> We couldn't log you in. Please try again.
curl -k -u elastic "https://localhost:9200/_cat/nodes/?v&h=name,du,dt,dup,hp,hc,rm,rp,r"
Enter host password for user 'elastic':
name du dt dup hp hc rm rp r
srvelk 91.2gb 95.8gb 95.12 33 2.6gb 15.6gb 96 cdfhilmrstw
root@srvelk:~# curl -k -u elastic "https://localhost:9200/_cat/indices/*?v&s=pri.store.size:desc"
Enter host password for user 'elastic':
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size dataset.size
yellow open .ds-filebeat-8.11.1-2023.12.16-000002 r4R7sGkZRPaCy4osrYAHxA 1 1 96630319 0 50gb 50gb 50gb
yellow open .ds-filebeat-8.11.1-2023.12.22-000003 5MaGlovySI-mU6ALskruoQ 1 1 27327097 0 14gb 14gb 14gb
yellow open .ds-filebeat-8.11.1-2023.11.16-000001 A7Y5ZlG7QheXryHgK0fQpA 1 1 22298154 0 12gb 12gb 12gb
green open .internal.alerts-observability.logs.alerts-default-000001 W-wiCs-7ToaRr9KjOjC0Iw 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.uptime.alerts-default-000001 NAS42JsdQbqhUTnzI6dvcw 1 0 0 0 249b 249b 249b
green open .internal.alerts-ml.anomaly-detection.alerts-default-000001 oe2borCxQ520NlD6Ok-Y2w 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.slo.alerts-default-000001 r8oGIwF-Rb-OEIHNuEPmcg 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.apm.alerts-default-000001 Wrgp26bbTlGTo6vYO3EOCQ 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.metrics.alerts-default-000001 iJ252YZ3Ty-ZWbx8MQihmA 1 0 0 0 249b 249b 249b
green open .kibana-observability-ai-assistant-conversations-000001 SbjXxyw9Q2u_WedT563dCw 1 0 0 0 249b 249b 249b
yellow open mikrotik a8eN4AFqTQicmAwB9qHmbg 1 1 0 0 249b 249b 249b
green open .internal.alerts-observability.threshold.alerts-default-000001 jLIF61xpTRieUynHu-YCMw 1 0 0 0 249b 249b 249b
green open .internal.alerts-security.alerts-default-000001 OoAsfZ4NQGWUTPRNXt1klw 1 0 0 0 249b 249b 249b
green open .kibana-observability-ai-assistant-kb-000001 wGU1lhc3ReGtaCQMYNKdkw 1 0 0 0 249b 249b 249b
green open .internal.alerts-stack.alerts-default-000001 XTeuNnW1RECTf797ioMpoQ 1 0 0 0 249b 249b 249b
It looks like you are ingesting data from filebeat into the cluster.... The data seems to be filling up pretty fast so even if you clean up you are going to back in the same state / issues soon. It looks like you will only be able to keep perhaps 7-10 days of data...
There are 2 Basic things that you are going to do.
- Clean up some space by deleting some indices
- Then, Set up an Index Lifecycle Management to automatically clean up the indices.
So First, you can DELETE a couple of those indices BUT
WARNING if you DELETE the indices the DATA is Lost forever unless you have another backup of the data
This command would free up about 12GB of data
curl -X DELETE -k -u elastic "https://localhost:9200/.ds-filebeat-8.11.1-2023.11.16-000001"
This Command will DELETE up another 50GB Data, BUT again you will lose all the data between 2023.12.16 and 2023.12.22
WARNING if you DELETE the indices the DATA is Lost forever unless you have another backup of the data
curl -X DELETE -k -u elastic "https://localhost:9200/.ds-filebeat-8.11.1-2023.12.16-000002"
After you do that... you should be able to log in unless there are other issues with your kibana etc... '
If you get logged into Kibana
Kibana -> Stack Management -> Index Lifecycle Policy - Filebeat
Set to rollover in 1 day, Delete after 10 Days : Save
oh Stephen .... thanks my friend 
You helped me a lot
Thank you very much 
1 Like
yes my friend
thanks so much 
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.