CSV overwrite/ delete index

[FoRMaTC]

Hi,

I'm downloading a CSV from our ticketing system and I want to import that into elastic every 15 minutes.
However I want to start with a clean slate over and over again.
What i've done now is wrote a small perl script which is downloading the CSV, deleting the data in the index and importing the csv via logstash into elastic.
Is there a beter way todo this?

Cheers,

Bjorn

[scottdfedorov (Scott Fedorov)]

Hard to answer without some actual examples of what the content being indexed is...

Why are you putting everything in the same index? Why don't you just create a new index at each run?

Can you use index lifecycle management for this? Just set the parameters so that the index rolls over before you run the pipeline again, and be sure in the ilm policy that you have it delete indexes after rollover.

[FoRMaTC]

Hi Scott, i want to load data in order to create some piecharts and datatables.
See it like I want to know all open tickets in 2019.
I have read the ilm but i cannot find an example for this. How do I make the index rollover before running the pipeline?

[scottdfedorov (Scott Fedorov)]

Still too vague to be able to help here. There may be better solutions but to use ILM on data that is loaded every 15 minutes, just make the index rollover before the 15 mins are up...

[FoRMaTC]

@scottdfedorov

Still too vague to be able to help here. There may be better solutions but to use ILM on data that is loaded every 15 minutes, just make the index rollover before the 15 mins are up...

See the csv like this:
status;ticketnr;engineer;datestart;comments
open;111;john;01-12-2019;blahblah
open;112;harry;02-12-2019;blahblah
atcustomer;113;peter;02-14-2019;blahblah
open;114;maria;03-12-2020;blahblah

I'm trying now to set a ILM up:

    "ticket_policy" : {
      "version" : 6,
      "modified_date" : "2020-03-19T11:05:26.817Z",
      "policy" : {
        "phases" : {
          "hot" : {
            "min_age" : "0ms",
            "actions" : {
              "rollover" : {
                "max_age" : "10m"
              }
            }
          },
          "delete" : {
            "min_age" : "1nanos",
            "actions" : {
              "delete" : { }
            }
          }
        }
      }
    }