How To delete/migrate old data from an index

Elastic Stack Elasticsearch
1

Hi people !

My situation: I have 2 logstash-forwarders sending network devices logs to an ELK server. In the past I did create the needed indexes based on time ( YYYY+MM+DD), but I detected that the performance of the server goes down with this config, and i changed it to static indexes, the performance problem was resolved with this change.

But, now i have big indexes on the elk server and i need retrieve data from them to migrate or delete it.
at the momment i was deleting the indexes and start them again from the scratch , but i want to have capability and the knowledge to select a specific info from an index and make different actions with it.

Thanks in advance

2

Two routes you could take:

  • Reindex has a "Reindex from Remote" functionality, that basically streams the data from one index to another on a different cluster. That will give you the flexibility to only select some data, subsets, certain types, etc

  • If you just want everything, Snapshot and Restore would be faster. But it's less flexible in what it can capture (e.g. it'll grab entire indices)

3

Hi @polyfractal, how are you ??

Thanks for your answer and sorry for take so much time to come back to this topic.

Is possible perform the same operation than " Reindex From Remote" but delete the data directly and not migrate it to another cluster ?
Is not my intention migrate the info, just delete it, but keeping the last 3 months.

Thanks In advance.

4

You can reindex into the same cluster, which allows you to move data from one index to another. You can then delete the old index if you want, or run a DeleteByQuery, etc.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.