Strange problem with curator

Elastic Stack Elasticsearch
1

I all,
i have a problem with the index deletion in ES made by curator.
I'm indexing my AS logs with the timestamp of the logs through logstash:

Logstash config:
input {
beats {
port => "5043"
}
}
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} "]
}
date {
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSS" ]
target => "@timestamp"
}
mutate {
remove_field => [ "timestamp" ]
}

}

When i run curator to delete the index of two days ago, logs from 00:00:00 to 02:00:00 og 1 day ago also been deleted.

Curator config:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 2 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 2

Please, can someone help me???

2

Logstash index names are based on UTC timestamp, so not necessarily aligned with your timezone. This is probably the reason you are seeing the described behaviour.

3

Ok great,
but can i try to solve the problem? Do you have some tips?

4

You are right:

from a check:
"@timestamp": "2017-09-13T09:33:46.472Z"
"message": "2017-09-13 11:33:46,472 DEBUG

How can i put the message timestamp in the @timestamp field?

5

Timestamps in Elasticsearch is always in UTC, so the example looks correct. If you want to base the index name on something other than the UTC timestamp, I suspect you will need to create the suffix yourself rather than rely on the timestamp pattern in the Elasticsearch output.

6

Please, can you help me to create this suffix?

7

You might be able to assemble it based on the components of the original timestamp extracted from the logs.

8

I don't know how to start...:frowning: this is my first installation. Sorry.....

9

It might be simpler to set the unit_count to 3. Just keep those indices a bit longer.

10

I write around 12 TB in a day....less log, less disk space..:slight_smile:

11

If you index that much per day and do not need to keep the data very long, you could consider switching to hourly indices with fewer primary shards.

12

Or using the Rollover API/Curator action so you can roll over you indices at a given size.

13

Ok,
now i launched a new test with 2 node, hourly index and 2 primary shards set.

For today log, ES has created:

15 indices, 76 total shards
for Documents: 65,357,155

I will have 90 days of retention...
With this number, my config is good??

14

In your initial example you had 2 days retention period, and here it might make sense to have hourly indices. For indices being kept for 90 days I would recommend instead using daily indices and dropping the indices a day later as this makes relatively small difference to the data volume stored.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.