# Strange problem with curator **URL:** \ **Category:** Elasticsearch\ **Created:** [September 13, 2017, 8:51am UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319 "2017-09-13T08:51:51Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![sbampa](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@sbampa](https://discuss.elastic.co/u/sbampa)\ **Post date:** [September 13, 2017, 8:51am UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/1 "2017-09-13T08:51:52Z")
I all, i have a problem with the index deletion in ES made by curator. I'm indexing my AS logs with the timestamp of the logs through logstash: Logstash config: input { beats { port =\> "5043" } } filter { grok { match =\> ["message", "%{TIMESTAMP\_ISO8601:timestamp} %{LOGLEVEL:level} "] } date { match =\> ["timestamp", "YYYY-MM-dd HH:mm:ss,SSS"] target =\> "@timestamp" } mutate { remove\_field =\> ["timestamp"] } } When i run curator to delete the index of two days ago, logs from 00:00:00 to 02:00:00 og 1 day ago also been deleted. Curator config: actions: 1: action: delete\_indices description: \>- Delete indices older than 2 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore\_empty\_list) and exit cleanly. options: ignore\_empty\_list: True disable\_action: False filters: - filtertype: pattern kind: prefix value: logstash- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit\_count: 2 Please, can someone help me??? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 13, 2017, 9:09am UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/2 "2017-09-13T09:09:02Z")
Logstash index names are based on UTC timestamp, so not necessarily aligned with your timezone. This is probably the reason you are seeing the described behaviour. ---
**Author:** ![sbampa](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@sbampa](https://discuss.elastic.co/u/sbampa)\ **Post date:** [September 13, 2017, 9:27am UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/3 "2017-09-13T09:27:22Z")
Ok great, but can i try to solve the problem? Do you have some tips? ---
**Author:** ![sbampa](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@sbampa](https://discuss.elastic.co/u/sbampa)\ **Post date:** [September 13, 2017, 9:38am UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/4 "2017-09-13T09:38:49Z")
You are right: from a check: "@timestamp": "2017-09-13T09:33:46.472Z" "message": "2017-09-13 11:33:46,472 DEBUG How can i put the message timestamp in the @timestamp field? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 13, 2017, 9:46am UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/5 "2017-09-13T09:46:08Z")
Timestamps in Elasticsearch is always in UTC, so the example looks correct. If you want to base the index name on something other than the UTC timestamp, I suspect you will need to create the suffix yourself rather than rely on the timestamp pattern in the Elasticsearch output. ---
**Author:** ![sbampa](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@sbampa](https://discuss.elastic.co/u/sbampa)\ **Post date:** [September 14, 2017, 1:34pm UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/6 "2017-09-14T13:34:33Z")
Please, can you help me to create this suffix? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 14, 2017, 1:58pm UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/7 "2017-09-14T13:58:07Z")
You might be able to assemble it based on the components of the original timestamp extracted from the logs. ---
**Author:** ![sbampa](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@sbampa](https://discuss.elastic.co/u/sbampa)\ **Post date:** [September 14, 2017, 2:10pm UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/8 "2017-09-14T14:10:24Z")
I don't know how to start...☹ this is my first installation. Sorry..... ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [September 14, 2017, 2:17pm UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/9 "2017-09-14T14:17:19Z")
It might be simpler to set the `unit_count` to 3. Just keep those indices a bit longer. ---
**Author:** ![sbampa](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@sbampa](https://discuss.elastic.co/u/sbampa)\ **Post date:** [September 14, 2017, 2:23pm UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/10 "2017-09-14T14:23:21Z")
I write around 12 TB in a day....less log, less disk space..🙂 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 14, 2017, 2:28pm UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/11 "2017-09-14T14:28:52Z")
If you index that much per day and do not need to keep the data very long, you could consider switching to hourly indices with fewer primary shards. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [September 14, 2017, 3:17pm UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/12 "2017-09-14T15:17:48Z")
Or using the Rollover API/Curator action so you can roll over you indices at a given size. ---
**Author:** ![sbampa](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@sbampa](https://discuss.elastic.co/u/sbampa)\ **Post date:** [September 15, 2017, 10:31am UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/13 "2017-09-15T10:31:52Z")
Ok, now i launched a new test with 2 node, hourly index and 2 primary shards set. For today log, ES has created: 15 indices, 76 total shards for Documents: 65,357,155 I will have 90 days of retention... With this number, my config is good?? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 15, 2017, 10:36am UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/14 "2017-09-15T10:36:36Z")
In your initial example you had 2 days retention period, and here it might make sense to have hourly indices. For indices being kept for 90 days I would recommend instead using daily indices and dropping the indices a day later as this makes relatively small difference to the data volume stored. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 13, 2017, 10:37am UTC](https://discuss.elastic.co/t/strange-problem-with-curator/100319/15 "2017-10-13T10:37:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.