# How to delete the old documnet from my index pattern?

**URL:** <https://discuss.elastic.co/t/how-to-delete-the-old-documnet-from-my-index-pattern/143376>\
**Category:** Elasticsearch\
**Created:** [August 7, 2018, 5:51pm UTC](https://discuss.elastic.co/t/how-to-delete-the-old-documnet-from-my-index-pattern/143376 "2018-08-07T17:51:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![vebabu](https://avatars.discourse-cdn.com/v4/letter/v/919ad9/32.png) [@vebabu](https://discuss.elastic.co/u/vebabu)\
**Post date:** [August 7, 2018, 5:51pm UTC](https://discuss.elastic.co/t/how-to-delete-the-old-documnet-from-my-index-pattern/143376/1 "2018-08-07T17:51:29Z")

</div>

Hi All,

I am collecting the data from IoT DEVICE every second and transferring the data to Elastic search. I wanted only last 7 days data in my index pattern. How do i delete the old document data from my Index pattern.

I have a field called "event\_ts" which is the time field and I want to perform the delete operation based on this time field.

**my data looks like :**

{  
"\_index": "dev-andon",  
"\_type": "dev-andon",  
"\_id": "2675ace7-b73c-4cea-b074-e1eb49fd6a1c",  
"\_version": 1,  
"\_score": 2,  
"\_source": {  
"raspi\_id": "0000000078686c63",  
"PK": "0000000078686c63\_inactive\_880d12a1-ac76-415e-9735-c57050319ccd\_2018-07-12",  
"activity\_id": "880d12a1-ac76-415e-9735-c57050319ccd\_2018-07-12",  
"event\_ts": "2018-07-12T16:26:09.505710",  
"event\_type": "inactive",  
"device": "knight",  
"activity": "andon"  
},  
"fields": {  
"event\_ts": [  
"2018-07-12T16:26:09.505Z"  
]  
}  
}

Query that I have tried.  
DELETE dev-andon/dev-andon/\_query  
{  
"query": {  
"filtered": {  
"query": {  
"query\_string": {  
"query": "\*"  
}  
},  
"filter": {  
"range": {  
"event\_ts": {  
"lte": "2018-07-13T01:00:00.000000"  
}  
}  
}  
}  
}  
}

The output I am getting :  
{  
"\_index": "dev-andon",  
"\_type": "dev-andon",  
"\_id": "\_query",  
"\_version": 1,  
"result": "not\_found",  
"\_shards": {  
"total": 2,  
"successful": 2,  
"failed": 0  
},  
"\_seq\_no": 47,  
"\_primary\_term": 1  
}

Please advice  
Best,  
Venkatesh

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 7, 2018, 7:26pm UTC](https://discuss.elastic.co/t/how-to-delete-the-old-documnet-from-my-index-pattern/143376/2 "2018-08-07T19:26:48Z")

</div>

Create time based indices like one index per day.  
At the end of the week, just drop the old indices.

You can automate that by using Curator.

---

<div class="post-metadata">

**Author:** ![vebabu](https://avatars.discourse-cdn.com/v4/letter/v/919ad9/32.png) [@vebabu](https://discuss.elastic.co/u/vebabu)\
**Post date:** [August 8, 2018, 1:45pm UTC](https://discuss.elastic.co/t/how-to-delete-the-old-documnet-from-my-index-pattern/143376/3 "2018-08-08T13:45:41Z")

</div>

Hi Dadoonet,

I don't want to create a Index pattern for every day , because I am build a dashboard based on the index pattern and I can't keep on build my dashboard. Is there any way i can search all the record in my index pattern and delete the record based on the time parameter ??

Please advice.  
Best,  
Venkatesh

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 8, 2018, 2:04pm UTC](https://discuss.elastic.co/t/how-to-delete-the-old-documnet-from-my-index-pattern/143376/4 "2018-08-08T14:04:18Z")

</div>

> [@vebabu](#):
>
> I don't want to create a Index pattern for every day

Of course. That's where index templates are great for.

On Kibana side, you can define the index pattern as `foo-*` for example.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 8, 2018, 3:01pm UTC](https://discuss.elastic.co/t/how-to-delete-the-old-documnet-from-my-index-pattern/143376/5 "2018-08-08T15:01:34Z")

</div>

@dadoonet is correct. There is no reason not to use a different index per day for your use case. Because Kibana can use an Index Pattern for building visualizations and dashboards, it can handle new indices per day:

![43%20AM](https://us1.discourse-cdn.com/elastic/original/3X/2/1/21f9099bf3b51d9acb18b19febd3c8327a624119.png)

If you click on Index Patterns, you will be taken to another screen where you can Create Index Pattern. You can see from mine that there are several Index Patterns which follow the `foo-*` example shared by @dadoonet:

![55%20AM](https://us1.discourse-cdn.com/elastic/original/3X/1/3/13b24691f6e55ad849383852ef074339555c0a3a.png)

Once created, when you go to create a new visualization in Kibana, your Index Pattern will be in a list you select from:

![38%20AM](https://us1.discourse-cdn.com/elastic/original/3X/9/6/96cb2ba17aa6712f09fe5bbc51569c29d9f0abd1.png)

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 8, 2018, 3:11pm UTC](https://discuss.elastic.co/t/how-to-delete-the-old-documnet-from-my-index-pattern/143376/6 "2018-08-08T15:11:42Z")

</div>

To further round out the discussion, performing delete\_by\_query, which you can still do, is very inefficient for deleting data from indices, compared with deleting entire indices. The difference is similar to the difference between these SQL psuedo-statements of:

```auto
DELETE from TABLE where timestamp < now-7days

```

and

```auto
DROP TABLE

```

The `DELETE from` statement has to perform a query, and do a comparison on every document, and then set up a series of atomic DELETE operations for each match found, while the DROP statement is over and done in a single shot. This example isn't perfect, because SQL is designed to handle this sort of thing better than Elasticsearch.

Elasticsearch makes things worse still because deleting a document doesn't result in it immediately freeing resources, but instead only _marks_ the document for deletion. It isn't actually deleted until the next segment merge—which results in yet another scan of the documents to see which are kept, and which to delete. That's at least 2 scans over all of your documents just to free the resources. Also, having mismatched segment sizes—which is what happens with document deletes—makes Lucene a bit less efficient.

Elasticsearch handles these scenarios well, but if you didn't have to delete documents from an index, it would be much, much more efficient, which is why @dadoonet recommended using daily indices—as do I.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2018, 3:11pm UTC](https://discuss.elastic.co/t/how-to-delete-the-old-documnet-from-my-index-pattern/143376/7 "2018-09-05T15:11:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.