Deleting old ES logs /var/log/elasticsearch


(Kasper Brandenburg) #1

Hi guys,

Help me configure log-retention for ES.
I've struggling to delete old logfiles created by my ES clusters.

I've tried several settings in logging.yml, and none of them (log4j) seems
to work.
The the file appenders i've tried to add MaxDateRollBackups, maxSizeRollBackups
an others, no result.

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
MaxDateRollBackups: 2
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

logfile format looks like this:

-rw-r--r-- 1 elasticsearch elasticsearch 29M Oct 30 23:59
SHQA01ESCL1.log.2014-10-30
-rw-r--r-- 1 elasticsearch elasticsearch 18M Oct 31 23:59
SHQA01ESCL1.log.2014-10-31
-rw-r--r-- 1 elasticsearch elasticsearch 13M Nov 1 23:58
SHQA01ESCL1.log.2014-11-01
-rw-r--r-- 1 elasticsearch elasticsearch 202M Nov 2 23:51
SHQA01ESCL1.log.2014-11-02
-rw-r--r-- 1 elasticsearch elasticsearch 35M Nov 3 23:59
SHQA01ESCL1.log.2014-11-03
-rw-r--r-- 1 elasticsearch elasticsearch 43M Nov 4 23:59
SHQA01ESCL1.log.2014-11-04
-rw-r--r-- 1 elasticsearch elasticsearch 4.6M Nov 5 19:33
SHQA01ESCL1.log.2014-11-05
-rw-r--r-- 1 elasticsearch elasticsearch 95M Nov 6 23:59
SHQA01ESCL1.log.2014-11-06

Currently my ES cluster is 1.0.3, and we should upgrade to latest version
in Q1, 2015..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c36959ae-951b-4cbc-ac5c-ef87ce26039e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Delete Old Log Files in ES 2.3.4
(Jörg Prante) #2

The dailyRollingFIle appender parameter for retention is 'maxBackupIndex'

E.g.

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
maxBackupIndex: 7
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Jörg

On Mon, Dec 8, 2014 at 12:57 PM, Kasper Brandenburg <
kasper.brandenburg@gmail.com> wrote:

Hi guys,

Help me configure log-retention for ES.
I've struggling to delete old logfiles created by my ES clusters.

I've tried several settings in logging.yml, and none of them (log4j) seems
to work.
The the file appenders i've tried to add MaxDateRollBackups, maxSizeRollBackups
an others, no result.

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
MaxDateRollBackups: 2
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

logfile format looks like this:

-rw-r--r-- 1 elasticsearch elasticsearch 29M Oct 30 23:59
SHQA01ESCL1.log.2014-10-30
-rw-r--r-- 1 elasticsearch elasticsearch 18M Oct 31 23:59
SHQA01ESCL1.log.2014-10-31
-rw-r--r-- 1 elasticsearch elasticsearch 13M Nov 1 23:58
SHQA01ESCL1.log.2014-11-01
-rw-r--r-- 1 elasticsearch elasticsearch 202M Nov 2 23:51
SHQA01ESCL1.log.2014-11-02
-rw-r--r-- 1 elasticsearch elasticsearch 35M Nov 3 23:59
SHQA01ESCL1.log.2014-11-03
-rw-r--r-- 1 elasticsearch elasticsearch 43M Nov 4 23:59
SHQA01ESCL1.log.2014-11-04
-rw-r--r-- 1 elasticsearch elasticsearch 4.6M Nov 5 19:33
SHQA01ESCL1.log.2014-11-05
-rw-r--r-- 1 elasticsearch elasticsearch 95M Nov 6 23:59
SHQA01ESCL1.log.2014-11-06

Currently my ES cluster is 1.0.3, and we should upgrade to latest version
in Q1, 2015..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c36959ae-951b-4cbc-ac5c-ef87ce26039e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c36959ae-951b-4cbc-ac5c-ef87ce26039e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGCwh7JzrV%3D%3DPEMfjKW2858mRQVaoTppov7FGjQgERVfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Log File Retention Settings
(Kasper Brandenburg) #3

The maxBackupIndex setting does not work on my version.
I've decided to use a simple find and run it as a cronjob instead.
Hopfully once i get the ES software upgraded

/usr/bin/find /var/log/elasticsearch/ -mtime +3 -type f -name '.log.20'
-exec /bin/rm -f '{}' +

Den mandag den 8. december 2014 12.57.55 UTC+1 skrev Kasper Brandenburg:

Hi guys,

Help me configure log-retention for ES.
I've struggling to delete old logfiles created by my ES clusters.

I've tried several settings in logging.yml, and none of them (log4j) seems
to work.
The the file appenders i've tried to add MaxDateRollBackups, maxSizeRollBackups
an others, no result.

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
MaxDateRollBackups: 2
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

logfile format looks like this:

-rw-r--r-- 1 elasticsearch elasticsearch 29M Oct 30 23:59
SHQA01ESCL1.log.2014-10-30
-rw-r--r-- 1 elasticsearch elasticsearch 18M Oct 31 23:59
SHQA01ESCL1.log.2014-10-31
-rw-r--r-- 1 elasticsearch elasticsearch 13M Nov 1 23:58
SHQA01ESCL1.log.2014-11-01
-rw-r--r-- 1 elasticsearch elasticsearch 202M Nov 2 23:51
SHQA01ESCL1.log.2014-11-02
-rw-r--r-- 1 elasticsearch elasticsearch 35M Nov 3 23:59
SHQA01ESCL1.log.2014-11-03
-rw-r--r-- 1 elasticsearch elasticsearch 43M Nov 4 23:59
SHQA01ESCL1.log.2014-11-04
-rw-r--r-- 1 elasticsearch elasticsearch 4.6M Nov 5 19:33
SHQA01ESCL1.log.2014-11-05
-rw-r--r-- 1 elasticsearch elasticsearch 95M Nov 6 23:59
SHQA01ESCL1.log.2014-11-06

Currently my ES cluster is 1.0.3, and we should upgrade to latest version
in Q1, 2015..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/31cec286-56b3-4220-9a6c-5ef5759b76f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Nik Everett) #4

Why not logrotate?

On Wed, Jan 7, 2015 at 3:25 AM, Kasper Brandenburg <
kasper.brandenburg@gmail.com> wrote:

The maxBackupIndex setting does not work on my version.
I've decided to use a simple find and run it as a cronjob instead.
Hopfully once i get the ES software upgraded

/usr/bin/find /var/log/elasticsearch/ -mtime +3 -type f -name '.log.20'
-exec /bin/rm -f '{}' +

Den mandag den 8. december 2014 12.57.55 UTC+1 skrev Kasper Brandenburg:

Hi guys,

Help me configure log-retention for ES.
I've struggling to delete old logfiles created by my ES clusters.

I've tried several settings in logging.yml, and none of them (log4j)
seems to work.
The the file appenders i've tried to add MaxDateRollBackups, maxSizeRollBackups
an others, no result.

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
MaxDateRollBackups: 2
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

logfile format looks like this:

-rw-r--r-- 1 elasticsearch elasticsearch 29M Oct 30 23:59
SHQA01ESCL1.log.2014-10-30
-rw-r--r-- 1 elasticsearch elasticsearch 18M Oct 31 23:59
SHQA01ESCL1.log.2014-10-31
-rw-r--r-- 1 elasticsearch elasticsearch 13M Nov 1 23:58
SHQA01ESCL1.log.2014-11-01
-rw-r--r-- 1 elasticsearch elasticsearch 202M Nov 2 23:51
SHQA01ESCL1.log.2014-11-02
-rw-r--r-- 1 elasticsearch elasticsearch 35M Nov 3 23:59
SHQA01ESCL1.log.2014-11-03
-rw-r--r-- 1 elasticsearch elasticsearch 43M Nov 4 23:59
SHQA01ESCL1.log.2014-11-04
-rw-r--r-- 1 elasticsearch elasticsearch 4.6M Nov 5 19:33
SHQA01ESCL1.log.2014-11-05
-rw-r--r-- 1 elasticsearch elasticsearch 95M Nov 6 23:59
SHQA01ESCL1.log.2014-11-06

Currently my ES cluster is 1.0.3, and we should upgrade to latest version
in Q1, 2015..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/31cec286-56b3-4220-9a6c-5ef5759b76f0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/31cec286-56b3-4220-9a6c-5ef5759b76f0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd29owgGWXoxwVtKh6niWhgCR_hv2iRHYN9bqxCeb%2Bcjkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #5