Space is getting filled. Need some information how to clear/re-claim the space


(Tamal Kundu) #1

Hi ,

We have Elasticserch for monitoring our services for business production. But the space is getting filled after a while. We are facing this issue quite often. Need to know if there is a way we can claim or clear the space, which logs we can clear.

[fuseadmin@a0110pcsgmon03 ~]$ df -kh
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/rhel_vg-root_lv   33G  1.9G   29G   6% /
devtmpfs                      32G     0   32G   0% /dev
tmpfs                         32G     0   32G   0% /dev/shm
tmpfs                         32G  449M   31G   2% /run
tmpfs                         32G     0   32G   0% /sys/fs/cgroup
/dev/sda1                    488M  147M  306M  33% /boot
/dev/mapper/rhel_vg-home_lv   30G  160M   28G   1% /home
/dev/mapper/rhel_vg-var_lv    20G  1.3G   18G   7% /var
/dev/mapper/app_vg-app_lv    1.5T 1011G  391G  73% /app
tmpfs                        6.3G     0  6.3G   0% /run/user/1009
[fuseadmin@a0110pcsgmon03 ~]$ du -sh *
4.7M    httpd_rpm
0       nohup.out
106M    openssl-1.0.2i
5.1M    openssl-1.0.2i.tar.gz
[fuseadmin@a0110pcsgmon03 ~]$

(Tamal Kundu) #2

it will very helpful if we can get a reply soon and also let me know you have any query,


(David Pilato) #3

Read this and specifically the "Also be patient" part.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.


(David Pilato) #4

Which space is filled? Does not look obvious to me.


(Tamal Kundu) #5

/app directory is getting filled. The log generation is huge and elastic search stopped after it reach to 85%. Today it reached to 79%.

[fuseadmin@a0110pcsgmon03 ~]$ df -kh
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/rhel_vg-root_lv   33G  1.9G   29G   6% /
devtmpfs                      32G     0   32G   0% /dev
tmpfs                         32G     0   32G   0% /dev/shm
tmpfs                         32G  449M   31G   2% /run
tmpfs                         32G     0   32G   0% /sys/fs/cgroup
/dev/sda1                    488M  147M  306M  33% /boot
/dev/mapper/rhel_vg-home_lv   30G  160M   28G   1% /home
/dev/mapper/rhel_vg-var_lv    20G  1.3G   18G   7% /var
/dev/mapper/app_vg-app_lv    1.5T  1.1T  300G  79% /app
tmpfs                        6.3G     0  6.3G   0% /run/user/1009
[fuseadmin@a0110pcsgmon03 ~]$ df -kh | grep -i /app
/dev/mapper/app_vg-app_lv    1.5T  1.1T  300G  79% /app
[fuseadmin@a0110pcsgmon03 ~]$

Let me know if way to delete some old logs or way to analysis more deathly.


(David Pilato) #6

Please format your code. I did it twice for you but please do it to make it easier to read. The preview window will help a lot. You can use markdown style.

So /app is where you put your elasticsearch data, right?

What gives:

GET _cat/nodes?v
GET _cat/indices?v

(Tamal Kundu) #7

yes. location is .

[fuseadmin@a0110pcsgmon03 elasticsearch-5.5.0]$ pwd
/app/elasticsearch-5.5.0
[fuseadmin@a0110pcsgmon03 elasticsearch-5.5.0]$
[fuseadmin@a0110pcsgmon03 elasticsearch-5.5.0]$ ll
total 256
drwxr-xr-x. 2 fuseadmin fueadmin 4096 Apr 26 18:59 bin
drwxr-xr-x. 3 fuseadmin fueadmin 4096 Apr 26 15:40 config
drwxr-xr-x. 3 fuseadmin fueadmin 4096 Sep 23 2017 data
drwxr-xr-x. 2 fuseadmin fueadmin 4096 Jul 1 2017 lib
-rw-r--r--. 1 fuseadmin fueadmin 11358 Jul 1 2017 LICENSE.txt
drwxr-xr-x. 2 fuseadmin fueadmin 16384 Apr 27 00:14 logs
drwxr-xr-x. 13 fuseadmin fueadmin 4096 Jul 1 2017 modules
-rw-------. 1 fuseadmin fueadmin 0 Apr 26 18:58 nohup.out
-rw-r--r--. 1 fuseadmin fueadmin 194187 Jul 1 2017 NOTICE.txt
drwxr-xr-x. 2 fuseadmin fueadmin 4096 Jul 1 2017 plugins
-rw-r--r--. 1 fuseadmin fueadmin 9548 Jul 1 2017 README.textile

You mentioned -"Please format your code. I did it twice for you but please do it to make it easier to read. "

Please let me know where and also let me know
GET _cat/nodes?v
GET _cat/indices?v


(David Pilato) #8

I did that for your first posts.

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.


(David Pilato) #9

Please provide what is the output of the cat API I asked for.


(Tamal Kundu) #10

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1 41 93 13 0.76 0.61 0.57 mdi * DiGiNode..


(Tamal Kundu) #11
yellow open   fuselog-2018.04.07 qkUqviZmTe6F4MhG_7Cxjg   5   1    3168101            0     16.4gb         16.4gb
yellow open   fuselog-2018.04.21 gF-izQo4RT-QNBxcO9HkFQ   5   1   15487399            0       58gb           58gb
yellow open   fuselog-2018.04.09 ySwGtyXCR7aO4yFQ4DEQDg   5   1    3209478            0     16.7gb         16.7gb
yellow open   fuselog-2018.04.18 muSCHehkRR62rugWr1TEEA   5   1    8280513            0     42.6gb         42.6gb
yellow open   fuselog-2018.02.04 PJLLWc4RQZWxpOYlTQqBFw   5   1      40367            0      123mb          123mb
yellow open   fuselog-2017.10.23 gZ5vPqyPSC-OuZXoRRM5jw   5   1         68            0      286kb          286kb
yellow open   fuselog-2018.03.07 BKYQfiCOQ9yEgGJoQxYUiA   5   1      85264            0    245.6mb        245.6mb
yellow open   fuselog-2018.03.17 AaPjQr2YRhCuTTFaJ4vjPA   5   1      70310            0    168.3mb        168.3mb
yellow open   fuselog-2017.11.06 UBLfwR4HTTyL03bHEdeFtA   5   1      13489            0     52.5mb         52.5mb
yellow open   fuselog-2018.02.07 Fvt2Z_IJSVmrJ0s0M1eseA   5   1      76747            0    185.9mb        185.9mb
yellow open   fuselog-2018.01.10 56G8VssWQwKt5FxpIg6QYQ   5   1      67867            0      232mb          232mb
yellow open   fuselog-2018.04.25 2VzrP0m6RuCwtsld55bIOw   5   1   20058747            0     73.8gb         73.8gb
yellow open   fuselog-2017.10.04 C03WwyPWTTKgwXyVSJEZhw   5   1        465            0    889.4kb        889.4kb
yellow open   fuselog-2017.11.07 B0ZDrowPTDWCm_KNMwiALQ   5   1      12790            0       50mb           50mb
yellow open   fuselog-2017.11.26 bQNnutBqRbGWfTHfkEHdaw   5   1      11520            0     44.7mb         44.7mb
yellow open   fuselog-2018.02.23 2nvpCBWwT9OVfGzYOnyhXg   5   1      75475            0    184.4mb        184.4mb
yellow open   fuselog-2018.01.15 tsjljB8OTOmCUpKZgGcO8w   5   1      77502            0    260.2mb        260.2mb
yellow open   fuselog-2018.01.02 udJnR03XQOy6TBceMteKLA   5   1      19767            0     75.1mb         75.1mb
yellow open   fuselog-2017.11.09 MJVD92JpRSyao3ZdUnQQfA   5   1      11273            0     43.4mb         43.4mb
yellow open   fuselog-2018.02.02 zljRRzlPQhqhNUrX69g2HA   5   1      37740            0    112.4mb        112.4mb
yellow open   fuselog-2018.01.01 opuxEctbQqGBB4LyKJs33A   5   1      22198            0     84.9mb         84.9mb
yellow open   fuselog-2018.02.01 YFRLdxdUQfKfWEOVdjeBBw   5   1      37182            0    117.7mb        117.7mb
yellow open   fuselog-2018.01.13 e_WXVw-lSVGzezySrJO8YQ   5   1     140233            0    449.8mb        449.8mb
yellow open   fuselog-2018.04.14 O4Cs4zLXRvqgwzfeKLfy9w   5   1    5916013            0     31.1gb         31.1gb
yellow open   fuselog-2017.11.23 o25xcTXcQV6biL04Gez1Rg   5   1     110738            0    126.6mb        126.6mb
yellow open   fuselog-2018.04.20 NGGZkWc8QiC7EzmN4TFUPg   5   1   15995158            0     57.9gb         57.9gb
yellow open   fuselog-2017.11.19 Rrzz3VEbTH-gFqC4nhdKTA   5   1      11206            0     44.5mb         44.5mb
yellow open   fuselog-2017.10.30 4nPdUpfKSySCb6bO-RZyzg   5   1      13549            0     52.6mb         52.6mb
yellow open   .kibana            fU0pVNukQG6VJi8dFBCD-g   1   1         32            2     82.2kb         82.2kb
yellow open   fuselog-2017.11.04 6qQ0l9jjTN2qpJ8o4WlCQg   5   1      14564            0     56.1mb         56.1mb
yellow open   fuselog-2018.04.22 ctBUjBEOSkOchcNUYxHT3Q   5   1   15670490            0     58.5gb         58.5gb

(David Pilato) #12

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.


(Tamal Kundu) #13
ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1           10          91  40    4.80    4.56     4.49 mdi       *      DiGiNode
    indent preformatted text by 4 spaces

(Tamal Kundu) #14
health status index              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   fuselog-2018.05.03 Ivek1mr5S9q1KiRDu-2NUg   5   1   12269800            0     49.5gb         49.5gb
yellow open   .kibana            MtbGHB3TSz6KivnuRarZQQ   1   1         31            0    108.1kb        108.1kb`Preformatted text`

(Tamal Kundu) #15

Any update


(David Pilato) #16

Are you sure you still need the old indices from 2017 ?

Anyway, apparently since the end of April you are injecting a lot of data.

What is the sum of all space that the _cat API is showing? Is it something like 1100 gb?
If not, run may be:

du -s /app

To see which dir is taking most of the space.


(Tamal Kundu) #17

[fuseadmin@a0110pcsgmon04 ~]$ du -sh --time /app
du: cannot read directory â/app/lost+foundâ: Permission denied
513G 2018-05-07 20:56 /app
[fuseadmin@a0110pcsgmon04 ~]$


(David Pilato) #18

Can you find which dir under app is taking all space? Did you sum as I asked the size given by the cat API?


(Tamal Kundu) #19

GET /_cat/allocation?v

shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
26 504gb 580.5gb 895.8gb 1.4tb 39 localhost 127.0.0.1 DiGiNode
26

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open fuselog-2018.05.05 pGGh2mwASGmHET_YKSgldA 5 1 28867041 0 103.4gb 103.4gb
yellow open fuselog-2018.05.07 UQvSXBKlSjeJM_bngM01wQ 5 1 35098404 0 124.6gb 124.6gb
yellow open data AlnpmM8QSeiFciLtJTkDCA 5 1 0 0 810b 810b
yellow open fuselog-2018.05.06 GqrrGFuvT56cWfCdEjcO8g 5 1 30833617 0 112.2gb 112.2gb
yellow open .kibana MtbGHB3TSz6KivnuRarZQQ 1 1 32 2 80.6kb 80.6kb
yellow open fuselog-2018.05.04 Cq1Cluf8SEK6F2wv3NFNGw 5 1 45942884 0 164.8gb 164.8gb

[fuseadmin@a0110pcsgmon04 elasticsearch-5.5.0]$ du -sh --time *
1.1M 2018-05-07 21:25 bin
24K 2018-05-03 19:16 config
520G 2018-05-07 21:25 data
22M 2017-07-01 07:20 lib
12K 2017-07-01 07:14 LICENSE.txt
824K 2018-05-07 21:25 logs
14M 2017-07-01 07:20 modules
192K 2017-07-01 07:20 NOTICE.txt
4.0K 2017-07-01 07:20 plugins
12K 2017-07-01 07:14 README.textile
[fuseadmin@a0110pcsgmon04 elasticsearch-5.5.0]$

[fuseadmin@a0110pcsgmon04 nodes]$ du -sh --time *
520G 2018-05-07 21:26 0
12K 2018-05-07 20:51 1
[fuseadmin@a0110pcsgmon04 nodes]$

[fuseadmin@a0110pcsgmon04 nodes]$ cd 0 && du -sh --time *
521G 2018-05-07 21:26 indices
0 2017-09-13 20:11 node.lock
12K 2018-05-07 20:35 _state
[fuseadmin@a0110pcsgmon04 0]$


(Tamal Kundu) #20

Please let me know if your require more any more output