Deleting old ES logs /var/log/elasticsearch

Kasper_Brandenburg · December 8, 2014, 11:57am

Hi guys,

Help me configure log-retention for ES.
I've struggling to delete old logfiles created by my ES clusters.

I've tried several settings in logging.yml, and none of them (log4j) seems
to work.
The the file appenders i've tried to add MaxDateRollBackups, maxSizeRollBackups
an others, no result.

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
MaxDateRollBackups: 2
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

logfile format looks like this:

-rw-r--r-- 1 elasticsearch elasticsearch 29M Oct 30 23:59
SHQA01ESCL1.log.2014-10-30
-rw-r--r-- 1 elasticsearch elasticsearch 18M Oct 31 23:59
SHQA01ESCL1.log.2014-10-31
-rw-r--r-- 1 elasticsearch elasticsearch 13M Nov 1 23:58
SHQA01ESCL1.log.2014-11-01
-rw-r--r-- 1 elasticsearch elasticsearch 202M Nov 2 23:51
SHQA01ESCL1.log.2014-11-02
-rw-r--r-- 1 elasticsearch elasticsearch 35M Nov 3 23:59
SHQA01ESCL1.log.2014-11-03
-rw-r--r-- 1 elasticsearch elasticsearch 43M Nov 4 23:59
SHQA01ESCL1.log.2014-11-04
-rw-r--r-- 1 elasticsearch elasticsearch 4.6M Nov 5 19:33
SHQA01ESCL1.log.2014-11-05
-rw-r--r-- 1 elasticsearch elasticsearch 95M Nov 6 23:59
SHQA01ESCL1.log.2014-11-06

Currently my ES cluster is 1.0.3, and we should upgrade to latest version
in Q1, 2015..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c36959ae-951b-4cbc-ac5c-ef87ce26039e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · December 8, 2014, 1:14pm

The dailyRollingFIle appender parameter for retention is 'maxBackupIndex'

E.g.

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
maxBackupIndex: 7
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Jörg

On Mon, Dec 8, 2014 at 12:57 PM, Kasper Brandenburg <
kasper.brandenburg@gmail.com> wrote:

Hi guys,

Help me configure log-retention for ES.
I've struggling to delete old logfiles created by my ES clusters.

I've tried several settings in logging.yml, and none of them (log4j) seems
to work.
The the file appenders i've tried to add MaxDateRollBackups, maxSizeRollBackups
an others, no result.

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
MaxDateRollBackups: 2
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

logfile format looks like this:

-rw-r--r-- 1 elasticsearch elasticsearch 29M Oct 30 23:59
SHQA01ESCL1.log.2014-10-30
-rw-r--r-- 1 elasticsearch elasticsearch 18M Oct 31 23:59
SHQA01ESCL1.log.2014-10-31
-rw-r--r-- 1 elasticsearch elasticsearch 13M Nov 1 23:58
SHQA01ESCL1.log.2014-11-01
-rw-r--r-- 1 elasticsearch elasticsearch 202M Nov 2 23:51
SHQA01ESCL1.log.2014-11-02
-rw-r--r-- 1 elasticsearch elasticsearch 35M Nov 3 23:59
SHQA01ESCL1.log.2014-11-03
-rw-r--r-- 1 elasticsearch elasticsearch 43M Nov 4 23:59
SHQA01ESCL1.log.2014-11-04
-rw-r--r-- 1 elasticsearch elasticsearch 4.6M Nov 5 19:33
SHQA01ESCL1.log.2014-11-05
-rw-r--r-- 1 elasticsearch elasticsearch 95M Nov 6 23:59
SHQA01ESCL1.log.2014-11-06

Currently my ES cluster is 1.0.3, and we should upgrade to latest version
in Q1, 2015..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c36959ae-951b-4cbc-ac5c-ef87ce26039e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c36959ae-951b-4cbc-ac5c-ef87ce26039e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGCwh7JzrV%3D%3DPEMfjKW2858mRQVaoTppov7FGjQgERVfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Kasper_Brandenburg · January 7, 2015, 8:25am

The maxBackupIndex setting does not work on my version.
I've decided to use a simple find and run it as a cronjob instead.
Hopfully once i get the ES software upgraded

/usr/bin/find /var/log/elasticsearch/ -mtime +3 -type f -name '.log.20'
-exec /bin/rm -f '{}' +

Den mandag den 8. december 2014 12.57.55 UTC+1 skrev Kasper Brandenburg:

Hi guys,

Help me configure log-retention for ES.
I've struggling to delete old logfiles created by my ES clusters.

I've tried several settings in logging.yml, and none of them (log4j) seems
to work.
The the file appenders i've tried to add MaxDateRollBackups, maxSizeRollBackups
an others, no result.

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
MaxDateRollBackups: 2
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

logfile format looks like this:

-rw-r--r-- 1 elasticsearch elasticsearch 29M Oct 30 23:59
SHQA01ESCL1.log.2014-10-30
-rw-r--r-- 1 elasticsearch elasticsearch 18M Oct 31 23:59
SHQA01ESCL1.log.2014-10-31
-rw-r--r-- 1 elasticsearch elasticsearch 13M Nov 1 23:58
SHQA01ESCL1.log.2014-11-01
-rw-r--r-- 1 elasticsearch elasticsearch 202M Nov 2 23:51
SHQA01ESCL1.log.2014-11-02
-rw-r--r-- 1 elasticsearch elasticsearch 35M Nov 3 23:59
SHQA01ESCL1.log.2014-11-03
-rw-r--r-- 1 elasticsearch elasticsearch 43M Nov 4 23:59
SHQA01ESCL1.log.2014-11-04
-rw-r--r-- 1 elasticsearch elasticsearch 4.6M Nov 5 19:33
SHQA01ESCL1.log.2014-11-05
-rw-r--r-- 1 elasticsearch elasticsearch 95M Nov 6 23:59
SHQA01ESCL1.log.2014-11-06

Currently my ES cluster is 1.0.3, and we should upgrade to latest version
in Q1, 2015..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/31cec286-56b3-4220-9a6c-5ef5759b76f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

nik9000 · January 7, 2015, 2:09pm

Why not logrotate?

On Wed, Jan 7, 2015 at 3:25 AM, Kasper Brandenburg <
kasper.brandenburg@gmail.com> wrote:

The maxBackupIndex setting does not work on my version.
I've decided to use a simple find and run it as a cronjob instead.
Hopfully once i get the ES software upgraded

/usr/bin/find /var/log/elasticsearch/ -mtime +3 -type f -name '.log.20'
-exec /bin/rm -f '{}' +

Den mandag den 8. december 2014 12.57.55 UTC+1 skrev Kasper Brandenburg:

Hi guys,

Help me configure log-retention for ES.
I've struggling to delete old logfiles created by my ES clusters.

I've tried several settings in logging.yml, and none of them (log4j)
seems to work.
The the file appenders i've tried to add MaxDateRollBackups, maxSizeRollBackups
an others, no result.

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
MaxDateRollBackups: 2
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

logfile format looks like this:

-rw-r--r-- 1 elasticsearch elasticsearch 29M Oct 30 23:59
SHQA01ESCL1.log.2014-10-30
-rw-r--r-- 1 elasticsearch elasticsearch 18M Oct 31 23:59
SHQA01ESCL1.log.2014-10-31
-rw-r--r-- 1 elasticsearch elasticsearch 13M Nov 1 23:58
SHQA01ESCL1.log.2014-11-01
-rw-r--r-- 1 elasticsearch elasticsearch 202M Nov 2 23:51
SHQA01ESCL1.log.2014-11-02
-rw-r--r-- 1 elasticsearch elasticsearch 35M Nov 3 23:59
SHQA01ESCL1.log.2014-11-03
-rw-r--r-- 1 elasticsearch elasticsearch 43M Nov 4 23:59
SHQA01ESCL1.log.2014-11-04
-rw-r--r-- 1 elasticsearch elasticsearch 4.6M Nov 5 19:33
SHQA01ESCL1.log.2014-11-05
-rw-r--r-- 1 elasticsearch elasticsearch 95M Nov 6 23:59
SHQA01ESCL1.log.2014-11-06

Currently my ES cluster is 1.0.3, and we should upgrade to latest version
in Q1, 2015..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/31cec286-56b3-4220-9a6c-5ef5759b76f0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/31cec286-56b3-4220-9a6c-5ef5759b76f0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd29owgGWXoxwVtKh6niWhgCR_hv2iRHYN9bqxCeb%2Bcjkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Delete Old Log Files in ES 2.3.4 Elasticsearch	2	1389	February 4, 2017
Log File Retention Settings Elasticsearch	4	17789	July 5, 2017
Elastic Search Logs Retention Elasticsearch	6	7092	July 6, 2017
ElasticSearch log management Elasticsearch	4	376	July 6, 2017
Pruning logs - how to get maxBackupIndex to work? Elasticsearch	2	2473	July 6, 2017

Deleting old ES logs /var/log/elasticsearch

Related topics