Logstash doesn't take changes from log4j2.properties after restart service

Elastic Stack Logstash
1

I've noticed that after service restart logstash doesn't apply rules modified in log4j2.properties i.e appender.rolling.policies.size.size = 10MB (I've checked and the file has 12MB so it should work)
Generally speaking, the goal is to delete logs older than 7D.

appender.rolling.type = RollingFile
appender.rolling.name = plain_rolling
appender.rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
appender.rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd-HH-mm}-%i.log.gz
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size = 10MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.action.basepath = ${sys:ls.logs}
appender.rolling.strategy.action.condition.type = IfLastModified
appender.rolling.strategy.action.condition.age = 7D
appender.rolling.strategy.action.PathConditions.type = IfFileName
appender.rolling.strategy.action.PathConditions.glob = logstash-${sys:ls.log.format}-*.gz
appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.rolling.strategy.action.condition.nested_condition.exceeds = 50MB

appender.rolling.avoid_pipelined_filter.type = ScriptFilter
appender.rolling.avoid_pipelined_filter.script.type = Script
appender.rolling.avoid_pipelined_filter.script.name = filter_no_pipelined
appender.rolling.avoid_pipelined_filter.script.language = JavaScript
appender.rolling.avoid_pipelined_filter.script.scriptText = ${sys:ls.pipeline.separate_logs} == false || !(logEvent.getContextData().containsKey("pipeline.id"))

The version of logstash and OS:
apt list --installed |grep logstash
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
logstash/bionic,stable,now 1:7.9.1-1 all [installed,upgradable to: 1:7.11.1-1]

lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

2

You may find this thread useful. Especially the part about the log4j2 properties being applied to the wrong appender.

1 Like
3

@Badger thx for help

the working configuration is below.
As you mentioned , appender responsible for deleting the apriopriate logs (logstash-plain-.log.gz) is json_rolling not rolling
The following configuration deletes logs ".gz" older than 7D or when their cumulative size is bigger then 100 MB.

appender.json_rolling.type = RollingFile
appender.json_rolling.name = json_rolling
appender.json_rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
appender.json_rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}-%i.log.gz
appender.json_rolling.policies.type = Policies
appender.json_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.json_rolling.policies.time.interval = 1
appender.json_rolling.policies.time.modulate = true
appender.json_rolling.layout.type = JSONLayout
appender.json_rolling.layout.compact = true
appender.json_rolling.layout.eventEol = true
appender.json_rolling.policies.size.type = SizeBasedTriggeringPolicy

appender.json_rolling.policies.size.size = 100MB
appender.json_rolling.strategy.type = DefaultRolloverStrategy
appender.json_rolling.strategy.max = 30
appender.json_rolling.strategy.action.type = Delete
appender.json_rolling.strategy.action.basepath = ${sys:ls.logs}
appender.json_rolling.strategy.action.condition.type = IfFileName
appender.json_rolling.strategy.action.condition.glob = logstash-${sys:ls.log.format}-*.gz
appender.json_rolling.strategy.action.ifAny.type = IfAny
appender.json_rolling.strategy.action.ifAny.ifLastModified.type = IfLastModified
appender.json_rolling.strategy.action.ifAny.ifLastModified.age = 7D
appender.json_rolling.strategy.action.ifAny.ifAccumulatedFileSize.type = IfAccumulatedFileSize
appender.json_rolling.strategy.action.ifAny.ifAccumulatedFileSize.exceeds = 100MB
1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.