Hi ,
We have Elasticserch for monitoring our services for business production. But the space is getting filled after a while. We are facing this issue quite often. Need to know if there is a way we can claim or clear the space, which logs we can clear.
[fuseadmin@a0110pcsgmon03 ~]$ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel_vg-root_lv 33G 1.9G 29G 6% /
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 449M 31G 2% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda1 488M 147M 306M 33% /boot
/dev/mapper/rhel_vg-home_lv 30G 160M 28G 1% /home
/dev/mapper/rhel_vg-var_lv 20G 1.3G 18G 7% /var
/dev/mapper/app_vg-app_lv 1.5T 1011G 391G 73% /app
tmpfs 6.3G 0 6.3G 0% /run/user/1009
[fuseadmin@a0110pcsgmon03 ~]$ du -sh *
4.7M httpd_rpm
0 nohup.out
106M openssl-1.0.2i
5.1M openssl-1.0.2i.tar.gz
[fuseadmin@a0110pcsgmon03 ~]$
it will very helpful if we can get a reply soon and also let me know you have any query,
dadoonet
April 26, 2018, 4:53pm
3
Read this and specifically the "Also be patient" part.
It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.
dadoonet
April 26, 2018, 4:55pm
4
Which space is filled? Does not look obvious to me.
/app directory is getting filled. The log generation is huge and elastic search stopped after it reach to 85%. Today it reached to 79%.
[fuseadmin@a0110pcsgmon03 ~]$ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel_vg-root_lv 33G 1.9G 29G 6% /
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 449M 31G 2% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda1 488M 147M 306M 33% /boot
/dev/mapper/rhel_vg-home_lv 30G 160M 28G 1% /home
/dev/mapper/rhel_vg-var_lv 20G 1.3G 18G 7% /var
/dev/mapper/app_vg-app_lv 1.5T 1.1T 300G 79% /app
tmpfs 6.3G 0 6.3G 0% /run/user/1009
[fuseadmin@a0110pcsgmon03 ~]$ df -kh | grep -i /app
/dev/mapper/app_vg-app_lv 1.5T 1.1T 300G 79% /app
[fuseadmin@a0110pcsgmon03 ~]$
Let me know if way to delete some old logs or way to analysis more deathly.
dadoonet
April 27, 2018, 8:46am
6
Please format your code. I did it twice for you but please do it to make it easier to read. The preview window will help a lot. You can use markdown style.
So /app is where you put your elasticsearch data, right?
What gives:
GET _cat/nodes?v
GET _cat/indices?v
yes. location is .
[fuseadmin@a0110pcsgmon03 elasticsearch-5.5.0]$ pwd
You mentioned -"Please format your code. I did it twice for you but please do it to make it easier to read. "
Please let me know where and also let me know
dadoonet
April 27, 2018, 11:34am
8
I did that for your first posts.
Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.
Or use markdown style like:
```
CODE
```
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
dadoonet
April 27, 2018, 11:34am
9
Please provide what is the output of the cat API I asked for.
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
yellow open fuselog-2018.04.07 qkUqviZmTe6F4MhG_7Cxjg 5 1 3168101 0 16.4gb 16.4gb
yellow open fuselog-2018.04.21 gF-izQo4RT-QNBxcO9HkFQ 5 1 15487399 0 58gb 58gb
yellow open fuselog-2018.04.09 ySwGtyXCR7aO4yFQ4DEQDg 5 1 3209478 0 16.7gb 16.7gb
yellow open fuselog-2018.04.18 muSCHehkRR62rugWr1TEEA 5 1 8280513 0 42.6gb 42.6gb
yellow open fuselog-2018.02.04 PJLLWc4RQZWxpOYlTQqBFw 5 1 40367 0 123mb 123mb
yellow open fuselog-2017.10.23 gZ5vPqyPSC-OuZXoRRM5jw 5 1 68 0 286kb 286kb
yellow open fuselog-2018.03.07 BKYQfiCOQ9yEgGJoQxYUiA 5 1 85264 0 245.6mb 245.6mb
yellow open fuselog-2018.03.17 AaPjQr2YRhCuTTFaJ4vjPA 5 1 70310 0 168.3mb 168.3mb
yellow open fuselog-2017.11.06 UBLfwR4HTTyL03bHEdeFtA 5 1 13489 0 52.5mb 52.5mb
yellow open fuselog-2018.02.07 Fvt2Z_IJSVmrJ0s0M1eseA 5 1 76747 0 185.9mb 185.9mb
yellow open fuselog-2018.01.10 56G8VssWQwKt5FxpIg6QYQ 5 1 67867 0 232mb 232mb
yellow open fuselog-2018.04.25 2VzrP0m6RuCwtsld55bIOw 5 1 20058747 0 73.8gb 73.8gb
yellow open fuselog-2017.10.04 C03WwyPWTTKgwXyVSJEZhw 5 1 465 0 889.4kb 889.4kb
yellow open fuselog-2017.11.07 B0ZDrowPTDWCm_KNMwiALQ 5 1 12790 0 50mb 50mb
yellow open fuselog-2017.11.26 bQNnutBqRbGWfTHfkEHdaw 5 1 11520 0 44.7mb 44.7mb
yellow open fuselog-2018.02.23 2nvpCBWwT9OVfGzYOnyhXg 5 1 75475 0 184.4mb 184.4mb
yellow open fuselog-2018.01.15 tsjljB8OTOmCUpKZgGcO8w 5 1 77502 0 260.2mb 260.2mb
yellow open fuselog-2018.01.02 udJnR03XQOy6TBceMteKLA 5 1 19767 0 75.1mb 75.1mb
yellow open fuselog-2017.11.09 MJVD92JpRSyao3ZdUnQQfA 5 1 11273 0 43.4mb 43.4mb
yellow open fuselog-2018.02.02 zljRRzlPQhqhNUrX69g2HA 5 1 37740 0 112.4mb 112.4mb
yellow open fuselog-2018.01.01 opuxEctbQqGBB4LyKJs33A 5 1 22198 0 84.9mb 84.9mb
yellow open fuselog-2018.02.01 YFRLdxdUQfKfWEOVdjeBBw 5 1 37182 0 117.7mb 117.7mb
yellow open fuselog-2018.01.13 e_WXVw-lSVGzezySrJO8YQ 5 1 140233 0 449.8mb 449.8mb
yellow open fuselog-2018.04.14 O4Cs4zLXRvqgwzfeKLfy9w 5 1 5916013 0 31.1gb 31.1gb
yellow open fuselog-2017.11.23 o25xcTXcQV6biL04Gez1Rg 5 1 110738 0 126.6mb 126.6mb
yellow open fuselog-2018.04.20 NGGZkWc8QiC7EzmN4TFUPg 5 1 15995158 0 57.9gb 57.9gb
yellow open fuselog-2017.11.19 Rrzz3VEbTH-gFqC4nhdKTA 5 1 11206 0 44.5mb 44.5mb
yellow open fuselog-2017.10.30 4nPdUpfKSySCb6bO-RZyzg 5 1 13549 0 52.6mb 52.6mb
yellow open .kibana fU0pVNukQG6VJi8dFBCD-g 1 1 32 2 82.2kb 82.2kb
yellow open fuselog-2017.11.04 6qQ0l9jjTN2qpJ8o4WlCQg 5 1 14564 0 56.1mb 56.1mb
yellow open fuselog-2018.04.22 ctBUjBEOSkOchcNUYxHT3Q 5 1 15670490 0 58.5gb 58.5gb
dadoonet
April 29, 2018, 8:51pm
12
Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.
Or use markdown style like:
```
CODE
```
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1 10 91 40 4.80 4.56 4.49 mdi * DiGiNode
indent preformatted text by 4 spaces
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open fuselog-2018.05.03 Ivek1mr5S9q1KiRDu-2NUg 5 1 12269800 0 49.5gb 49.5gb
yellow open .kibana MtbGHB3TSz6KivnuRarZQQ 1 1 31 0 108.1kb 108.1kb`Preformatted text`
dadoonet
May 4, 2018, 5:31pm
16
Are you sure you still need the old indices from 2017 ?
Anyway, apparently since the end of April you are injecting a lot of data.
What is the sum of all space that the _cat API is showing? Is it something like 1100 gb?
du -s /app
To see which dir is taking most of the space.
[fuseadmin@a0110pcsgmon04 ~]$ du -sh --time /app
dadoonet
May 7, 2018, 1:20pm
18
Can you find which dir under app is taking all space? Did you sum as I asked the size given by the cat API?
GET /_cat/allocation?v
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
[fuseadmin@a0110pcsgmon04 elasticsearch-5.5.0]$ du -sh --time *
[fuseadmin@a0110pcsgmon04 nodes]$ du -sh --time *
[fuseadmin@a0110pcsgmon04 nodes]$ cd 0 && du -sh --time *
Please let me know if your require more any more output