Develop an Elastic plugin for automatic index restore when you need them

Elastic Stack Elasticsearch
1

Hello

I'm using ElasticSearch 6.7.2 and I have a massive amount of documents organized on daily indexes. So, for financial reasons, I decided to snapshot the old indexes (with more than 9 months) in Google Cloud Storage and only restore them when it's needed.

I installed the GCS repository plugin https://www.elastic.co/guide/en/elasticsearch/plugins/master/repository-gcs.html and the snapshot part are working perfectly and automated using curator. Now, I need to restore my index when my user search for it (long waits are acceptable).

I decided build an Elastic's plugin. This plugin has to verify if the searched index exists or not on cluster. If not, restores it from Cloud Storage. I'm trying to do that with an ActionFilter class , "intercepting" the _search requests and perform the aforementioned steps.

But I can not check if the index exists in the database. How am I supposed to do that? An ActionFilter class is the correct to use on my case? I cannot find many documentation about build ElasticSearch plugins and many of tutorials I have been finding are deprecated for the newer versions.

Thanks for your attention.

2

Why not using frozen indices? https://www.elastic.co/guide/en/elasticsearch/reference/current/frozen-indices.html

Where is the cost you are speaking about ? Disk? Memory ?

3

Sorry, I should have mentioned it. We talk about disks cost. If I keep all my indices on ElasticSearch's cluster, I will have more or less 32TB of data. Performing the snapshot and restore as aforementioned, I will keep only 2TB of data online on my cluster.

4

What if you use spinning disks or attached slow disks and use https://www.elastic.co/guide/en/elasticsearch/reference/current/frozen-indices.html? That could take several minutes to execute the search but at least will remove the burden of writing your own plugin.

5

I already saw this solution. The problem is again the cost. Using spinning disks my cost will be, on a conservative estimate, 260 times great then the Cloud Storage Solution. So, I chose build the plugin to automated the restore process. I only need to know how test if an index exists or not and how perform the restore inside the Elastic's plugin code.

6

@rjernst might be able to tell if/how this is possible.

7

For future viewers, I was able to solve the problem Overriding the Plugin class method createComponents and get the Client object parameter. But as you can see in this post Create a plugin that needs a client isn't a beautiful way, but works.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.