# Disk space got full cause of error.log and shards are failing to initialize

**URL:** https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895
**Category:** Elasticsearch
**Created:** [November 23, 2017, 1:36pm UTC](https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895 "2017-11-23T13:36:17Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![Kostas](https://avatars.discourse-cdn.com/v4/letter/k/bc8723/32.png) [@Kostas](https://discuss.elastic.co/u/Kostas)
#### Post date: [November 23, 2017, 1:36pm UTC](https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895/1 "2017-11-23T13:36:17Z")

</div>

I'm new to elastic and i would appreciate any help. I have a problem that lies to the error.log file, which size has become enormous and the shards are failing to initialize.

Could you please direct me to the right direction in order to solve the problem and also to the way to take precaution in order for the problem to not appear again? I am in development environment right now but it would be great to take measures in order to avoid such problems when i go to production.

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 23, 2017, 1:57pm UTC](https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895/2 "2017-11-23T13:57:46Z")

</div>

The first thing to do IMO is to get the first lines of your logs when it started to fail

---

<div class="post-metadata">

### Author: ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)
#### Post date: [November 23, 2017, 3:38pm UTC](https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895/3 "2017-11-23T15:38:28Z")

</div>

From sysadmin's point of view I'll suggest:

- Activate logrotate by size and add logrotate into hourly cron
- Separate log partition from data one

---

<div class="post-metadata">

### Author: ![Kostas](https://avatars.discourse-cdn.com/v4/letter/k/bc8723/32.png) [@Kostas](https://discuss.elastic.co/u/Kostas)
#### Post date: [November 24, 2017, 1:41pm UTC](https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895/4 "2017-11-24T13:41:49Z")

</div>

First i compressed the error.log file and shards started working. Then i checked logrotate by size and i added a file with the text below :

/var/log/nginx/elasticsearch/error.log {  
maxsize 1G  
minsize 1G  
hourly  
rotate 1  
compress  
notifempty  
postrotate  
find /var/log/nginx/elasticsearch -name "error.log.\*.gz" -mtime +30 -delete  
endscript  
}

and as i can see it works perfectly. It compresses the error.log file if it's size is greater than 1G and this check is done every hour. Thank you very much Vitaly\_il!!

---

<div class="post-metadata">

### Author: ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)
#### Post date: [November 26, 2017, 6:57am UTC](https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895/5 "2017-11-26T06:57:13Z")

</div>

Kostas,  
my pleasure; but I strongly suggest to check the reason such big logs - it seems be either serious elastic issue or too high loglevel.  
Vitaly

---

<div class="post-metadata">

### Author: ![Kostas](https://avatars.discourse-cdn.com/v4/letter/k/bc8723/32.png) [@Kostas](https://discuss.elastic.co/u/Kostas)
#### Post date: [November 27, 2017, 2:09pm UTC](https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895/6 "2017-11-27T14:09:24Z")

</div>

It seems that every time i make a request to my site error.log fills up 200-400kb and when i read the log i see that 97% of the messages are "debug" type. Should i stop loggin these debug messages and how would i do that?? Are they important/usefull?

---

<div class="post-metadata">

### Author: ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)
#### Post date: [November 27, 2017, 2:46pm UTC](https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895/7 "2017-11-27T14:46:32Z")

</div>

I suggest to decrease loglevel to "INFO" or even "WARN".  
Usually it's in /etc/elasticsearch/log4j2.properties, see [https://www.elastic.co/guide/en/elasticsearch/reference/5.6/settings.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/settings.html) for more.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 25, 2017, 2:47pm UTC](https://discuss.elastic.co/t/disk-space-got-full-cause-of-error-log-and-shards-are-failing-to-initialize/108895/8 "2017-12-25T14:47:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
