Elasticsearch logs are getting piled up in /var/log/elasticsearch path. How to cater to this problem?

[Sujata_Yadav]

Hi,

I am using elasticsearch to analyze my jenkins console data. I have observed that the
/var/log/elasticsearch directory is piled up with elasticsearch-2019-XX-YY-1.log.gz logs. Where XX-month, YY-Day.

Can you please tell me how to remove these logs. Can I remove these logs using a cron job and elasticsearch will work perfectly post using that?

[thomas.heuberger]

Logrotate is your friend. Yes, everything will work. Logs are just that ... logs.

https://linux.die.net/man/8/logrotate

[vasek (Václav Šulc)]

Rotating of logs is done by Elasticsearch using /etc/elasticsearch/log4j2.properties.
You can remove files using logrotate utility too. I recommand using only one way.

There is documentation of logging reference of Elasticsearch.
https://www.elastic.co/guide/en/elasticsearch/reference/7.1/logging.html

Simulation
You can easily test whether rotation and deleting of your logs works.

Configuration of log4j2

vim /etc/elasticsearch/log4j2.properties

################################################
######## Server -  old style pattern ###########
appender.rolling_old.type = RollingFile
appender.rolling_old.name = rolling_old
appender.rolling_old.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log
appender.rolling_old.layout.type = PatternLayout
appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

appender.rolling_old.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
appender.rolling_old.policies.type = Policies
appender.rolling_old.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling_old.policies.time.interval = 1
appender.rolling_old.policies.time.modulate = true
appender.rolling_old.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling_old.policies.size.size = 1MB
appender.rolling_old.strategy.type = DefaultRolloverStrategy
appender.rolling_old.strategy.fileIndex = nomax
appender.rolling_old.strategy.action.type = Delete
appender.rolling_old.strategy.action.basepath = ${sys:es.logs.base_path}
appender.rolling_old.strategy.action.condition.type = IfFileName
appender.rolling_old.strategy.action.condition.glob = ${sys:es.logs.cluster_name}-*
appender.rolling_old.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.rolling_old.strategy.action.condition.nested_condition.exceeds = 2MB
################################################

Notice

appender.rolling_old.policies.size.size = 1MB
appender.rolling_old.strategy.action.condition.nested_condition.exceeds = 2MB

Applying configuration changes

systemctl restart elasticsearch

Change level of logging

curl -XPUT -H 'Content-Type: application/json' 'localhost:9200/_cluster/settings' -d '{
    "transient" : {
        "logger._root": "TRACE"
    }
}'

Watch the current file of file which will be rotated

watch -n1 'stat --printf="%s" /var/log/elasticsearch/local.logs.itles.cz.log'

Watch deleting of rotated files (*.gz files)

watch -n1 'ls -latrh /var/log/elasticsearch/local*gz'

local.logs.itles.cz-2019-06-17-60.log.gz
local.logs.itles.cz-2019-06-17-61.log.gz
local.logs.itles.cz-2019-06-17-62.log.gz
local.logs.itles.cz-2019-06-17-63.log.gz
local.logs.itles.cz-2019-06-17-64.log.gz
local.logs.itles.cz-2019-06-17-65.log.gz
local.logs.itles.cz-2019-06-17-66.log.gz
local.logs.itles.cz-2019-06-17-67.log.gz
local.logs.itles.cz-2019-06-17-68.log.gz

It works fine.

[vasek (Václav Šulc)]

In my opinion:

From the perspective of disk space usage:

Rotation/deleting based on size is more secure than rotation/deleting based on time.