# Purge index **URL:** \ **Category:** Elasticsearch\ **Created:** [August 7, 2023, 11:59am UTC](https://discuss.elastic.co/t/purge-index/340265 "2023-08-07T11:59:57Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\ **Post date:** [August 7, 2023, 11:59am UTC](https://discuss.elastic.co/t/purge-index/340265/1 "2023-08-07T11:59:57Z")
hello, I would like to purge my data from my indexes in elasticsearch. I'd like to know how to do this without having to delete my index. How can I achieve that? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 7, 2023, 1:21pm UTC](https://discuss.elastic.co/t/purge-index/340265/2 "2023-08-07T13:21:58Z")
How much do you want to purge? 1%? 70%? ---
**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\ **Post date:** [August 7, 2023, 1:24pm UTC](https://discuss.elastic.co/t/purge-index/340265/3 "2023-08-07T13:24:53Z")
I would like my data to be kept for 6 months. So I'd like to set up a life cycle. Data older than 6 months will be deleted automatically by elasticsearch. How do I go about this, knowing that I already have the index set up and a data view created from this index? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 7, 2023, 1:52pm UTC](https://discuss.elastic.co/t/purge-index/340265/4 "2023-08-07T13:52:02Z")
The best way is here: [ILM: Manage the index lifecycle | Elasticsearch Guide [8.9] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) ---
**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\ **Post date:** [August 7, 2023, 2:44pm UTC](https://discuss.elastic.co/t/purge-index/340265/5 "2023-08-07T14:44:53Z")
Can I have an exemple please? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 7, 2023, 3:00pm UTC](https://discuss.elastic.co/t/purge-index/340265/6 "2023-08-07T15:00:16Z")
There are a lot of examples in the documentation. For example on this page: [Configure a lifecycle policy | Elasticsearch Guide [8.9] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) ```auto PUT _ilm/policy/my_policy { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_primary_shard_size": "25GB" } } }, "delete": { "min_age": "30d", "actions": { "delete": {} } } } } } ``` So if I want to apply that to your question, that'd be: ```auto PUT _ilm/policy/delete_after_180_days { "policy": { "phases": { "delete": { "min_age": "180d", "actions": { "delete": {} } } } } } ``` But please note that this will delete the indices older than 180 days. Which is by far the best way to do that. What is your usecase? Timeseries data? What kind of data is that? Which tool is collecting/sending the data? ---
**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\ **Post date:** [August 7, 2023, 3:06pm UTC](https://discuss.elastic.co/t/purge-index/340265/7 "2023-08-07T15:06:33Z")
the problem is that I don't want to delete my index. I want to be able to delete documents from my index that are more than 6 months old. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 7, 2023, 3:14pm UTC](https://discuss.elastic.co/t/purge-index/340265/8 "2023-08-07T15:14:13Z")
That's a very bad design but I can tell you how to do it although **I DO NOT RECOMMEND** doing that. The [DeleteByQuery API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html) allows that. You just need to find the right query which selects the right documents. This depends on your use case and fields you have within you documents. But again, don't do this! ---
**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\ **Post date:** [August 8, 2023, 8:18am UTC](https://discuss.elastic.co/t/purge-index/340265/9 "2023-08-08T08:18:02Z")
If you don't recommend it, I'd rather not do it. I'd just want elasticsearch to remove certain documents from my index at some point to avoid my index being overloaded later. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 8, 2023, 8:23am UTC](https://discuss.elastic.co/t/purge-index/340265/10 "2023-08-08T08:23:09Z")
Is your data immutable or do you perform updates? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 8, 2023, 8:23am UTC](https://discuss.elastic.co/t/purge-index/340265/11 "2023-08-08T08:23:44Z")
If it's a small number of documents, that's ok. If it's an entire day because documents have expired, then use Index Lifecycle Management with one index per day for example (depending on the size of your data). Another nice solution is to use datastreams if you have append-only data. See [Data streams | Elasticsearch Guide [8.9] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html) ---
**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\ **Post date:** [August 8, 2023, 8:30am UTC](https://discuss.elastic.co/t/purge-index/340265/12 "2023-08-08T08:30:28Z")
So if I understand correctly I have to create a data stream corresponding to my index? Will it have an impact on my index? Because my index is fed from a logstash script. Then I created a data view from my index to be able to visualize it on kibana. So I'd like to know if using data feeds will have any impact on my architecture. ---
**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\ **Post date:** [August 8, 2023, 8:39am UTC](https://discuss.elastic.co/t/purge-index/340265/13 "2023-08-08T08:39:44Z")
Here is my logstash conf ```auto input { file { path => "/scanNessus/*.csv" start_position => "beginning" #sincedb_path => "/dev/null" } } filter { csv { separator => "," skip_header => "true" skip_empty_rows => true columns => [ "Plugin_ID", "CVE", "CVSS_v2.0_Base_Score", "Risk", "Host", "Protocol", "Port", "Name", "CVSS_v3.0_Base_Score", "CVSS_v2.0_Temporal_Score", "CVSS_v3.0_Temporal_Score", "Risk_Factor", "Metasploit", "Core_Impact", "CANVAS", "date", "Nom_Application", "Niveau_2_Contact", "Niveau_3_Contact" ] } mutate { add_field => { "Plugin_Host" => "%{Plugin_ID}-%{Host}" } } } output { elasticsearch { hosts => "localhost:9200" index => "nms" template_name => "template" } stdout {} } ``` All my data are sent in my index name nms ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 8, 2023, 9:14am UTC](https://discuss.elastic.co/t/purge-index/340265/14 "2023-08-08T09:14:40Z")
Are you trying to get information from [Welcome to Tenable Security Center (Tenable Security Center 6.2.x)](https://docs.tenable.com/security-center/Content/Welcome.htm)? If so, I think you should just use the integration we have: > **[Tenable.sc](https://docs.elastic.co/en/integrations/tenable_sc)** > > Collect logs from Tenable.sc with Elastic Agent. > **[Tenable Vulnerability Management](https://docs.elastic.co/integrations/tenable_io)** > > Collect logs from Tenable Vulnerability Management with Elastic Agent. > [@Hanni](#): > > So if I understand correctly I have to create a data stream corresponding to my index? If you are using an alias, [that will be easy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-a-data-stream.html#convert-index-alias-to-data-stream). If not, then you probably need to do everything from scratch. See [Set up a data stream | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-a-data-stream.html) But as noted: > If you use Fleet or Elastic Agent, skip this tutorial. Fleet and Elastic Agent set up data streams for you. See Fleet’s [data streams](https://www.elastic.co/guide/en/fleet/8.9/data-streams.html) documentation. Also, using [ECS](https://www.elastic.co/what-is/ecs) instead of your own schema for this use case is better IMO as it's a de-facto standard nowadays. This is what Elastic Agents are actually generating. > [@Hanni](#): > > So I'd like to know if using data feeds will have any impact on my architecture. Here's some documentation about Logstash datastreams: [Elasticsearch output plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 5, 2023, 9:14am UTC](https://discuss.elastic.co/t/purge-index/340265/15 "2023-09-05T09:14:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.