Encrypt Translog or completely disable it?

Hi all,

is there a possibility to disable the Translog ? I guess that not.

And if we would like to encrypt the Translog, how could we proceed ?

In an earlier version of ES, the Translog was using a TransLog - store that could be cusomtized. But this feature has been removed. Is there another way to achieve this ?
Could the entire Translog - Component be replaced by an aggregated "proxy" or something like that by means of Guice ? Or do you have another idea ?

We would have to encrypt the Translog on a per-index basis (key per index) and the key is not know in the OS, just passed along the REST calls in custom http - headers .
Any chace?

Thanks!

Martin

1 Like

What's the goal in doing this?

we want to ensure end-to-end privacy on an index level by encrypting the index with a user specific key .

We already managed to encrypt the Lucene index, but the translog files also hold the content of the indexed documents in plain, at least for a certain time.
Such an administrator or someone with access to the filesystem can read the files and such there wouldn't be privacy.

How are you doing this?

We implemented an ActionPlugin.
In the "onIndexModule" - Method we add a new IndexStore, that delivers a custom DirectoryFactory, that one a custom FSDirectory and that one is using encrypted Streams ... Our index template is configured so that this index store is used .

In the action filters of the plugin we intercept the HTTP headers sent to the node, that hold the key for the index and put it into a local cache (indexname to key) with 5 minutes eviction time. In the IOFactory we retrieve the key again from that cache.

Currently this only is failing in 2 situations:

  1. if a document update should be written into a replica shard on a non-master node, because then our ActionFilter is not called (I created another posting for this here)
  2. if the cluster is restarted and the index still is open => there we intend to close all indexes on shutdown and/or on startup (also another posting here) .

Do you have an idea regarding the Translog?

Or would it be possible, to re-merge the former funcationality to have a TranslogStore that could be customized ?

Cheers,
Martin

Elasticsearch was not designed to exist without a translog, so I don't think you will have any luck.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.