Kibana Displays status Red

[Ramakrishna_K (Ramakrishna Kommineni)]

Hi,

I have been using elk stack with metric beats to collect linux systems performance data from 28 servers. For the past few days I am able to get the stats & analyze data.

But today when I login I see the below status in Kibana:

Also I can see that the disk space is full because of the elasticsearch /var/lib/elasticsearch/nodes/0/indices

[root@elk-stack indices]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 392G 391G 64M 100% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 116M 7.7G 2% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda3 392G 391G 64M 100% /home
/dev/sda1 1014M 133M 882M 14% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/1000

My questions are below:

1. I don't need more than 4 days logs. How can I set logrotation mechanism to delete elasticsearch indices older than 3 days?

2. How to increase the heap size as shown in kibana screenshot?

Could you please provide the information or point me to the link for the documentation related to this?
Thanks in advance.

Regards,
Ramakrishna

[warkolm (Mark Walkom)]

1. Elasticsearch Curator - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html
2. You shouldn't need to.

[Ramakrishna_K (Ramakrishna Kommineni)]

Hi,

Thanks for the quick response. After installing the curator I have done the following:

1. Queried what index is present on elasticsearch.

curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana qMXNaZi3RGi9wLx_OSufuQ 1 1 103 2 199.8kb 199.8kb
yellow open metricbeat-2017.02.13 ZJIUNUlfSjub30P89rKjfw 5 1 129884924 0 43.4gb 43.4gb
yellow open metricbeat-2017.02.15 hXWRqdCqQP6qR3t3TsQRog 5 1 83957276 0 33.3gb 33.3gb
yellow open metricbeat-2017.02.14 4tBKsQEhSZGniKdRGv6Fdg 5 1 129916020 0 43.8gb 43.8gb
yellow open metricbeat-2017.02.12 7csYVn2fR4-9JojjuBuFlg 5 1 129874841 0 43.1gb 43.1gb
yellow open metricbeat-2017.02.16 -O9dibroRPyiTmW3SKE2cw 5 1 522998 0 219.8mb 219.8mb

2. Created the below action file for curator based on examples.

3. ran the below command to delete all metricbeat indices which are older than 5 days.

curator /root/actionfile.yml

Now only 5 days data is present & disk space became normal.

[root@elk-stack ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 392G 163G 228G 42% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 8.5M 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda3 392G 163G 228G 42% /home
/dev/sda1 1014M 133M 882M 14% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/1000
[root@elk-stack ~]#

Now I was able to display stats normally.

Many thanks for the support.

Regards,
Ramakrishna