# CSV overwrite/ delete index

**URL:** <https://discuss.elastic.co/t/csv-overwrite-delete-index/224196>\
**Category:** Logstash\
**Created:** [March 18, 2020, 10:41pm UTC](https://discuss.elastic.co/t/csv-overwrite-delete-index/224196 "2020-03-18T22:41:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![FoRMaTC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/formatc/32/41508_2.png) [@FoRMaTC](https://discuss.elastic.co/u/FoRMaTC)\
**Post date:** [March 18, 2020, 10:41pm UTC](https://discuss.elastic.co/t/csv-overwrite-delete-index/224196/1 "2020-03-18T22:41:56Z")

</div>

Hi,

I'm downloading a CSV from our ticketing system and I want to import that into elastic every 15 minutes.  
However I want to start with a clean slate over and over again.  
What i've done now is wrote a small perl script which is downloading the CSV, deleting the data in the index and importing the csv via logstash into elastic.  
Is there a beter way todo this?

Cheers,

Bjorn

---

<div class="post-metadata">

**Author:** ![scottdfedorov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scottdfedorov/32/64699_2.png) [@scottdfedorov](https://discuss.elastic.co/u/scottdfedorov)\
**Post date:** [March 19, 2020, 1:53am UTC](https://discuss.elastic.co/t/csv-overwrite-delete-index/224196/2 "2020-03-19T01:53:47Z")

</div>

Hard to answer without some actual examples of what the content being indexed is...

Why are you putting everything in the same index? Why don't you just create a new index at each run?

Can you use [index lifecycle management](https://www.elastic.co/guide/en/elasticsearch/reference/master/overview-index-lifecycle-management.html) for this? Just set the parameters so that the index rolls over before you run the pipeline again, and be sure in the ilm policy that you have it delete indexes after rollover.

---

<div class="post-metadata">

**Author:** ![FoRMaTC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/formatc/32/41508_2.png) [@FoRMaTC](https://discuss.elastic.co/u/FoRMaTC)\
**Post date:** [March 19, 2020, 9:11am UTC](https://discuss.elastic.co/t/csv-overwrite-delete-index/224196/3 "2020-03-19T09:11:44Z")

</div>

Hi Scott, i want to load data in order to create some piecharts and datatables.  
See it like I want to know all open tickets in 2019.  
I have read the ilm but i cannot find an example for this. How do I make the index rollover before running the pipeline?

---

<div class="post-metadata">

**Author:** ![scottdfedorov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scottdfedorov/32/64699_2.png) [@scottdfedorov](https://discuss.elastic.co/u/scottdfedorov)\
**Post date:** [March 19, 2020, 4:44pm UTC](https://discuss.elastic.co/t/csv-overwrite-delete-index/224196/4 "2020-03-19T16:44:19Z")

</div>

Still too vague to be able to help here. There may be better solutions but to use ILM on data that is loaded every 15 minutes, just make the index rollover before the 15 mins are up...

---

<div class="post-metadata">

**Author:** ![FoRMaTC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/formatc/32/41508_2.png) [@FoRMaTC](https://discuss.elastic.co/u/FoRMaTC)\
**Post date:** [March 20, 2020, 8:45am UTC](https://discuss.elastic.co/t/csv-overwrite-delete-index/224196/5 "2020-03-20T08:45:45Z")

</div>

> [@scottdfedorov](#):
>
> Still too vague to be able to help here. There may be better solutions but to use ILM on data that is loaded every 15 minutes, just make the index rollover before the 15 mins are up...

See the csv like this:  
status;ticketnr;engineer;datestart;comments  
open;111;john;01-12-2019;blahblah  
open;112;harry;02-12-2019;blahblah  
atcustomer;113;peter;02-14-2019;blahblah  
open;114;maria;03-12-2020;blahblah

I'm trying now to set a ILM up:

```auto
    "ticket_policy" : {
      "version" : 6,
      "modified_date" : "2020-03-19T11:05:26.817Z",
      "policy" : {
        "phases" : {
          "hot" : {
            "min_age" : "0ms",
            "actions" : {
              "rollover" : {
                "max_age" : "10m"
              }
            }
          },
          "delete" : {
            "min_age" : "1nanos",
            "actions" : {
              "delete" : { }
            }
          }
        }
      }
    }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 17, 2020, 8:45am UTC](https://discuss.elastic.co/t/csv-overwrite-delete-index/224196/6 "2020-04-17T08:45:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.