# Elasticsearch logs are getting piled up in /var/log/elasticsearch path. How to cater to this problem?

**URL:** <https://discuss.elastic.co/t/elasticsearch-logs-are-getting-piled-up-in-var-log-elasticsearch-path-how-to-cater-to-this-problem/186039>\
**Category:** Elasticsearch\
**Created:** [June 17, 2019, 10:19am UTC](https://discuss.elastic.co/t/elasticsearch-logs-are-getting-piled-up-in-var-log-elasticsearch-path-how-to-cater-to-this-problem/186039 "2019-06-17T10:19:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sujata\_Yadav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujata_yadav/32/41882_2.png) [@Sujata\_Yadav](https://discuss.elastic.co/u/Sujata_Yadav)\
**Post date:** [June 17, 2019, 10:19am UTC](https://discuss.elastic.co/t/elasticsearch-logs-are-getting-piled-up-in-var-log-elasticsearch-path-how-to-cater-to-this-problem/186039/1 "2019-06-17T10:19:21Z")

</div>

Hi,

I am using elasticsearch to analyze my jenkins console data. I have observed that the  
/var/log/elasticsearch directory is piled up with elasticsearch-2019-XX-YY-1.log.gz logs. Where XX-month, YY-Day.

Can you please tell me how to remove these logs. Can I remove these logs using a cron job and elasticsearch will work perfectly post using that?

---

<div class="post-metadata">

**Author:** ![thomas.heuberger](https://avatars.discourse-cdn.com/v4/letter/t/e8c25b/32.png) [@thomas.heuberger](https://discuss.elastic.co/u/thomas.heuberger)\
**Post date:** [June 17, 2019, 10:24am UTC](https://discuss.elastic.co/t/elasticsearch-logs-are-getting-piled-up-in-var-log-elasticsearch-path-how-to-cater-to-this-problem/186039/2 "2019-06-17T10:24:12Z")

</div>

Logrotate is your friend. Yes, everything will work. Logs are just that ... logs.

[https://linux.die.net/man/8/logrotate](https://linux.die.net/man/8/logrotate)

---

<div class="post-metadata">

**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\
**Post date:** [June 17, 2019, 10:26am UTC](https://discuss.elastic.co/t/elasticsearch-logs-are-getting-piled-up-in-var-log-elasticsearch-path-how-to-cater-to-this-problem/186039/3 "2019-06-17T10:26:08Z")

</div>

Rotating of logs is done by Elasticsearch using **/etc/elasticsearch/log4j2.properties**.  
You can remove files using **[logrotate](https://linuxconfig.org/logrotate-8-manual-page)** utility too. I recommand using only one way.

There is documentation of logging reference of Elasticsearch.  
[https://www.elastic.co/guide/en/elasticsearch/reference/7.1/logging.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.1/logging.html)

**Simulation**  
You can easily test whether **rotation** and **deleting** of your logs works.

**Configuration of log4j2**

```
vim /etc/elasticsearch/log4j2.properties

################################################
######## Server - old style pattern ###########
appender.rolling_old.type = RollingFile
appender.rolling_old.name = rolling_old
appender.rolling_old.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log
appender.rolling_old.layout.type = PatternLayout
appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

appender.rolling_old.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
appender.rolling_old.policies.type = Policies
appender.rolling_old.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling_old.policies.time.interval = 1
appender.rolling_old.policies.time.modulate = true
appender.rolling_old.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling_old.policies.size.size = 1MB
appender.rolling_old.strategy.type = DefaultRolloverStrategy
appender.rolling_old.strategy.fileIndex = nomax
appender.rolling_old.strategy.action.type = Delete
appender.rolling_old.strategy.action.basepath = ${sys:es.logs.base_path}
appender.rolling_old.strategy.action.condition.type = IfFileName
appender.rolling_old.strategy.action.condition.glob = ${sys:es.logs.cluster_name}-*
appender.rolling_old.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.rolling_old.strategy.action.condition.nested_condition.exceeds = 2MB
################################################

```

**Notice**

```
appender.rolling_old.policies.size.size = 1MB
appender.rolling_old.strategy.action.condition.nested_condition.exceeds = 2MB

```

**Applying configuration changes**

```
systemctl restart elasticsearch

```

**Change level of logging**

```
curl -XPUT -H 'Content-Type: application/json' 'localhost:9200/_cluster/settings' -d '{
    "transient" : {
        "logger._root": "TRACE"
    }
}'

```

**Watch the current file of file which will be rotated**

```
watch -n1 'stat --printf="%s" /var/log/elasticsearch/local.logs.itles.cz.log'

```

**Watch deleting of rotated files (\*.gz files)**

```
watch -n1 'ls -latrh /var/log/elasticsearch/local*gz'

local.logs.itles.cz-2019-06-17-60.log.gz
local.logs.itles.cz-2019-06-17-61.log.gz
local.logs.itles.cz-2019-06-17-62.log.gz
local.logs.itles.cz-2019-06-17-63.log.gz
local.logs.itles.cz-2019-06-17-64.log.gz
local.logs.itles.cz-2019-06-17-65.log.gz
local.logs.itles.cz-2019-06-17-66.log.gz
local.logs.itles.cz-2019-06-17-67.log.gz
local.logs.itles.cz-2019-06-17-68.log.gz

```

It works fine.

---

<div class="post-metadata">

**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\
**Post date:** [June 17, 2019, 12:11pm UTC](https://discuss.elastic.co/t/elasticsearch-logs-are-getting-piled-up-in-var-log-elasticsearch-path-how-to-cater-to-this-problem/186039/4 "2019-06-17T12:11:42Z")

</div>

In my opinion:

From the perspective of disk space usage:

> Rotation/deleting **based on size is more secure** than rotation/deleting based on time.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 15, 2019, 12:11pm UTC](https://discuss.elastic.co/t/elasticsearch-logs-are-getting-piled-up-in-var-log-elasticsearch-path-how-to-cater-to-this-problem/186039/5 "2019-07-15T12:11:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.