Modify core storage of Elastic search

Elastic Stack Elasticsearch
1

I am new to ES project and i am trying to understand the code. What am trying to do is instead of ES writing to DISK, i want to write to some other place. Ex amazon S3. I know that this is not ideal, but I have my own reasons for this.

Can someone point me to the code where i can start changing this? Any guidance will he helpful.

2

Elasticsearch is very I/O intensive, so using something like S3 as backing storage is not a good idea. It may help if you provide some additional details about exactly what you want to do and why.

3

I want to encrypt data the Elastic search stores. So basically i want to encrypt using different keys per index. If i use S3 as file system then i can use S3s encryption at rest support.

Coming back to my main question:
Can you point me out to the code where ES writes to filesystem?

4

S3 does not offer many of the filesystem features that Elasticsearch needs to operate correctly, such as locks or atomic renames. It's just not possible to do what you ask.

Encryption-at-rest is perfectly possible today, with a proper filesystem on a proper hard disk, using something like dm-crypt.

If you are interested in the enormity of the task you suggest, the three major places that Elasticsearch writes to disk are:

  • org.elasticsearch.gateway.MetaDataStateFormat and its subclasses
  • org.apache.lucene.store.Directory and its subclasses
  • org.elasticsearch.index.translog.TranslogWriter
1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.