Hey all,
My ES logs are growing quite large on a daily basis. I understand that ES uses log4j for log rotation. But unfortunately I don't know enough about it to rotate my logs as needed. Currently ES logs can grow up to 8GB and overwhelm the partition.
This is my log4j file definition in /etc/elasticsearch/logging.yml:
file: #type: dailyRollingFile type: file file: ${path.logs}/${cluster.name}.log datePattern: "'.'yyyy-MM-dd" layout: type: pattern conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %.10000m%n"
How can I alter this definition so that log files never grow past 1gb and are gzipped once it gets past that point. Also how do I set the def so that zipped logs older than 3 days old get deleted.
Here's my full logging.yml file in case that's a help:
`# you can override this using by setting a system property, for example -Des.logger.level=DEBUG
es.logger.level: INFO
rootLogger: ${es.logger.level}, console, file
logger:
log action execution errors for easier debugging
action: DEBUG
deprecation logging, turn to DEBUG to see them
deprecation: INFO, deprecation_log_file
reduce the logging for aws, too much is logged under the default INFO
com.amazonaws: WARN
aws will try to do some sketchy JMX stuff, but its not needed.
com.amazonaws.jmx.SdkMBeanRegistrySupport: ERROR
com.amazonaws.metrics.AwsSdkMetrics: ERROR
org.apache.http: INFO
gateway
#gateway: DEBUG
#index.gateway: DEBUG
peer shard recovery
#indices.recovery: DEBUG
discovery
#discovery: TRACE
index.search.slowlog: TRACE, index_search_slow_log_file
index.indexing.slowlog: TRACE, index_indexing_slow_log_file
additivity:
index.search.slowlog: false
index.indexing.slowlog: false
deprecation: false
appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
file:
#type: dailyRollingFile
type: file
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %.10000m%n"
Use the following log4j-extras RollingFileAppender to enable gzip compression of log files.
For more information see https://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html
#file:
#type: extrasRollingFile
#file: ${path.logs}/${cluster.name}.log
#rollingPolicy: timeBased
#rollingPolicy.FileNamePattern: ${path.logs}/${cluster.name}.log.%d{yyyy-MM-dd}.gz
#layout:
#type: pattern
#conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
deprecation_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_deprecation.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
index_search_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_search_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
index_indexing_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"`
Lastly if anyone can point me to a good log4j tutorial so I can have a better understanding of how this works, I'd appreciate that as well.