is there a possibility to disable the Translog ? I guess that not.
And if we would like to encrypt the Translog, how could we proceed ?
In an earlier version of ES, the Translog was using a TransLog - store that could be cusomtized. But this feature has been removed. Is there another way to achieve this ?
Could the entire Translog - Component be replaced by an aggregated "proxy" or something like that by means of Guice ? Or do you have another idea ?
We would have to encrypt the Translog on a per-index basis (key per index) and the key is not know in the OS, just passed along the REST calls in custom http - headers .
Any chace?
we want to ensure end-to-end privacy on an index level by encrypting the index with a user specific key .
We already managed to encrypt the Lucene index, but the translog files also hold the content of the indexed documents in plain, at least for a certain time.
Such an administrator or someone with access to the filesystem can read the files and such there wouldn't be privacy.
We implemented an ActionPlugin.
In the "onIndexModule" - Method we add a new IndexStore, that delivers a custom DirectoryFactory, that one a custom FSDirectory and that one is using encrypted Streams ... Our index template is configured so that this index store is used .
In the action filters of the plugin we intercept the HTTP headers sent to the node, that hold the key for the index and put it into a local cache (indexname to key) with 5 minutes eviction time. In the IOFactory we retrieve the key again from that cache.
Currently this only is failing in 2 situations:
if a document update should be written into a replica shard on a non-master node, because then our ActionFilter is not called (I created another posting for this here)
if the cluster is restarted and the index still is open => there we intend to close all indexes on shutdown and/or on startup (also another posting here) .
Do you have an idea regarding the Translog?
Or would it be possible, to re-merge the former funcationality to have a TranslogStore that could be customized ?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.