# Delete documents by timestamp **URL:** \ **Category:** Elasticsearch\ **Created:** [July 6, 2017, 9:04am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058 "2017-07-06T09:04:04Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [July 6, 2017, 9:04am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/1 "2017-07-06T09:04:04Z")
I am using ELK, and some of my indexes are getting large. I would like to delete some documents that fir provided timeframe. For example, delete all documents in certain time range. I am using 5.3.2 ElasticSearch I use Kibana to issue queries to ES. Thank you. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2017, 9:04am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/2 "2017-07-06T09:04:42Z")
You need to use [https://www.elastic.co/guide/en/elasticsearch/reference/5.4/docs-delete-by-query.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/docs-delete-by-query.html) unless you are using time based indices. ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [July 6, 2017, 9:09am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/3 "2017-07-06T09:09:37Z")
Each of my documents has a @timestamp when it was indexed. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 6, 2017, 9:38am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/4 "2017-07-06T09:38:45Z")
Are you using [time-based indices](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/time-based.html)? ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [July 6, 2017, 9:41am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/5 "2017-07-06T09:41:17Z")
I am sorry if I do not understand exactly, but I will try to answer 🙂 Whenever a log is written as document, a @timestamp is appended at the time of the log creation. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 6, 2017, 9:48am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/6 "2017-07-06T09:48:31Z")
If you are writing into a single index, and not using a time-based one, e.g. `logstash-2017.07.06`, you will need to use delete-by-query which Mark linked to. This is generally much less efficient than simply dropping whole indices if you are using indices per time period. ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [July 6, 2017, 9:51am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/7 "2017-07-06T09:51:22Z")
Yes. I am writing in to single index. ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [July 6, 2017, 10:17am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/8 "2017-07-06T10:17:17Z")
This is what I have come up with: > POST myindex/\_delete\_by\_query > { > "query": { > "range" : { > "@timestamp" : { > "gte" : "09/02/2017", > "lte" : "11/02/2017", > "format": "dd/MM/yyyy||yyyy" > } > } > } > } And it works. Thank you for your suggestions 🙂 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2017, 10:19am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/9 "2017-07-06T10:19:29Z")
You should really change to time based indices. ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [July 6, 2017, 10:23am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/10 "2017-07-06T10:23:58Z")
Just to recap if I understand it correctly. You suggest that I create an index per day worth of logs. For example, in my folder application generates logs with a datestamp, something like: mylog.log.2017-06-06. And then I should create an index with the same name. Doing that, I can simply delete indexes that I do not wish. Did I understand it correctly? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2017, 10:27am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/11 "2017-07-06T10:27:53Z")
Yep! Daily, weekly or monthly is better than a single index/ ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [July 6, 2017, 10:31am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/12 "2017-07-06T10:31:59Z")
I see your point. But if I were to make a search on some of the indexes, and I would like to make deeper (several months in the past) search, then would it not complicate my queries? If I have indexes based on days, than I would take all of them in to account. Or even if I have them broken by months, then also if I wish to do annual search, than I would need to group the somehow? Currently, I am using Kibana to show me some statistics for one year or so. Am I missing something? 🙂 Thanks ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2017, 10:33am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/13 "2017-07-06T10:33:42Z")
> [@Vedran\_Maricevic](#): > > then would it not complicate my queries? No, Kibana has intelligence built into it to take this into account and will only query the indices it needs based on the requested timestamps. ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [July 6, 2017, 10:35am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/14 "2017-07-06T10:35:29Z")
Wow, I did not know that. So I might have 365 indexes, which is a Year worth... and Kibana will simply ignore it? I mean figure it out and make whatever 'joins' internally? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2017, 10:36am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/15 "2017-07-06T10:36:58Z")
That's the high level idea, yes 🙂 ---
**Author:** ![Vedran\_Maricevic](https://avatars.discourse-cdn.com/v4/letter/v/57b2e6/32.png) [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Post date:** [July 6, 2017, 10:37am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/16 "2017-07-06T10:37:54Z")
Are there any other benefits to have many indexes vs one big? In terms of speed or performance? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2017, 10:38am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/17 "2017-07-06T10:38:45Z")
It's **much** easier to manage retention. But unless you have too many shards then there's no real difference. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 6, 2017, 10:42am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/18 "2017-07-06T10:42:10Z")
If your data volumes allow for a single index, I would probably recommend switching to monthly rather than daily indices. As data is allocated to indices based on timestamp, Kibana can limit the number of indices queried to only the ones that hold data relevant to that period, which can result in less data queried. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 3, 2017, 10:42am UTC](https://discuss.elastic.co/t/delete-documents-by-timestamp/92058/19 "2017-08-03T10:42:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.