Strange problem with curator


#1

I all,
i have a problem with the index deletion in ES made by curator.
I'm indexing my AS logs with the timestamp of the logs through logstash:

Logstash config:
input {
beats {
port => "5043"
}
}
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} "]
}
date {
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSS" ]
target => "@timestamp"
}
mutate {
remove_field => [ "timestamp" ]
}

}

When i run curator to delete the index of two days ago, logs from 00:00:00 to 02:00:00 og 1 day ago also been deleted.

Curator config:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 2 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 2

Please, can someone help me???


(Christian Dahlqvist) #2

Logstash index names are based on UTC timestamp, so not necessarily aligned with your timezone. This is probably the reason you are seeing the described behaviour.


#3

Ok great,
but can i try to solve the problem? Do you have some tips?


#4

You are right:

from a check:
"@timestamp": "2017-09-13T09:33:46.472Z"
"message": "2017-09-13 11:33:46,472 DEBUG

How can i put the message timestamp in the @timestamp field?


(Christian Dahlqvist) #5

Timestamps in Elasticsearch is always in UTC, so the example looks correct. If you want to base the index name on something other than the UTC timestamp, I suspect you will need to create the suffix yourself rather than rely on the timestamp pattern in the Elasticsearch output.


#6

Please, can you help me to create this suffix?


(Christian Dahlqvist) #7

You might be able to assemble it based on the components of the original timestamp extracted from the logs.


#8

I don't know how to start...:frowning: this is my first installation. Sorry.....


(Aaron Mildenstein) #9

It might be simpler to set the unit_count to 3. Just keep those indices a bit longer.


#10

I write around 12 TB in a day....less log, less disk space..:slight_smile:


(Christian Dahlqvist) #11

If you index that much per day and do not need to keep the data very long, you could consider switching to hourly indices with fewer primary shards.


(Aaron Mildenstein) #12

Or using the Rollover API/Curator action so you can roll over you indices at a given size.


#13

Ok,
now i launched a new test with 2 node, hourly index and 2 primary shards set.

For today log, ES has created:

15 indices, 76 total shards
for Documents: 65,357,155

I will have 90 days of retention...
With this number, my config is good??


(Christian Dahlqvist) #14

In your initial example you had 2 days retention period, and here it might make sense to have hourly indices. For indices being kept for 90 days I would recommend instead using daily indices and dropping the indices a day later as this makes relatively small difference to the data volume stored.


(system) #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.