Elasticsearch Hot-Warn-Cold on a single node!

Elastic Stack Elasticsearch
1

is there a way to archive old data on elasticsearch and automatically move it to cheaper storage while keeping access to this data from the kibana interface ?

Our architecture has a single node that run, logstash, elasticsearch and kibana.

I went through most of the solutions on the forum and the solutions offered are:

  1. With Curator which consists of snapshot / restore -> the solution does not suit me because I would like something automatic where I will not need to restore to consult the archived data.

  2. Implementing a Hot-Warm-Cold Architecture with Index Lifecycle Management which requires a minimum of 2 nodes -> I only have one node.

basically I would like to know if I can implement solution 2 with a single node?

Thank you in advance for your answers.

2

Hot/warm architectures are about having different nodes optimised for different phases. You can not move data between paths on a single node.

3

@Christian_Dahlqvist

Thank you for your answer, so for what I want to do I must necessarily have a second node?

Do you know of other solutions to advise me?

thank you

4

You do need multiple nodes. Am not aware of any other solution.

1 Like
5

@Julius
On an ES node you can specify one or more directories as data directories. But once specified, there is no way, at least I know of, to tell ES which shard goes to which directory. So if you use fast and slow (cheaper) storage as directories on a single ES node, there is no way to achieve the hot warm architecture.

There are couple of possibilities, each with limitations

  1. If your real constraint is a single host (physical machine) and the host has enough resources, you can run multiple ES nodes on a single host. One node can use fast storage and other can use cheaper storage. You can also adjust heap. If you use docker you will have more control on cpu memory used by each node.
  2. Freeze old indices. Frozen indices are read-only and do not consume resources. So queries will run significantly slower. But data will be searchable. You can unfreeze without going through restore process. It will help you on memory / CPU but not on storage cost.
1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.