# Enabling encrypted indexes **URL:** \ **Category:** Elasticsearch\ **Created:** [May 30, 2013, 12:39pm UTC](https://discuss.elastic.co/t/enabling-encrypted-indexes/12191 "2013-05-30T12:39:18Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Mattijs\_Ugen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattijs_ugen/32/2303_2.png) [@Mattijs\_Ugen](https://discuss.elastic.co/u/Mattijs_Ugen)\ **Post date:** [May 30, 2013, 12:39pm UTC](https://discuss.elastic.co/t/enabling-encrypted-indexes/12191/1 "2013-05-30T12:39:18Z")
Friends, I've been trying to get elasticsearch to store indexes in an encrypted format. I know people have opted it here, but using full disk encryption is not a solution for me. The only place where the information that is being indexed is allowed to be available in plain form is in memory. This allows elasticsearch/Lucene to do its thing while the actual information cannot be read from disk without the proper key material. As far as I can find, there's no such implementation for elasticsearch as of yet. What I've done so far, is create I/O wrappers for Lucene 4.3 that accept read/seek/write requests like regular IndexInput and IndexOutput instances, but use CipherInputStream and CipherOutputStream to delegate the byte operations to a 'real' IndexInput or IndexOutput. This seems to work fine. I've found that elasticsearch does a comparable trick with wrapping index I/O instances, so I figured my code should be relatively easy to apply to the elasticsearch code base. Alas, this turns out to be untrue, as finding where to insert my wrappers on top of / below elasticsearch's is proving problematic. Wrapping the entire directory instance in StoreFileMetaData with one that wraps all I/O requests with one of my wrappers is problematic as the tests never seem to create an empty index, wrapping the I/O classes in Store.StoreDirectory doesn't seem to work either as it keeps creating I/O classes outside this point, causing it to read encrypted data without decrypting it. So I'm looking for the best place to hook this into elasticsearch. I've verified that my encryption/decryption wrappers work with Lucene, so I'd like to build a POC to make this work within elasticsearch. Any great ideas from the community? πŸ™‚ Kind regards, Mattijs -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mike\_Powers](https://avatars.discourse-cdn.com/v4/letter/m/ee7513/32.png) [@Mike\_Powers](https://discuss.elastic.co/u/Mike_Powers)\ **Post date:** [November 6, 2013, 3:37am UTC](https://discuss.elastic.co/t/enabling-encrypted-indexes/12191/2 "2013-11-06T03:37:52Z")
Hi Mattijs, Did you ever come up with a solution for encrypting ES indexes? - It seems that similar questions have been asked a few times, but I haven't found anyone who has developed a solution? Thanks! -Mike On Thursday, May 30, 2013 8:39:18 AM UTC-4, Mattijs Ugen wrote: > Friends, > > I've been trying to get elasticsearch to store indexes in an encrypted > format. I know people have opted it here, but using full disk encryption is > not a solution for me. The only place where the information that is being > indexed is allowed to be available in plain form is in memory. This allows > elasticsearch/Lucene to do its thing while the actual information cannot be > read from disk without the proper key material. As far as I can find, > there's no such implementation for elasticsearch as of yet. > > What I've done so far, is create I/O wrappers for Lucene 4.3 that accept > read/seek/write requests like regular IndexInput and IndexOutput instances, > but use CipherInputStream and CipherOutputStream to delegate the byte > operations to a 'real' IndexInput or IndexOutput. This seems to work fine. > I've found that elasticsearch does a comparable trick with wrapping index > I/O instances, so I figured my code should be relatively easy to apply to > the elasticsearch code base. > > Alas, this turns out to be untrue, as finding where to insert my wrappers > on top of / below elasticsearch's is proving problematic. Wrapping the > entire directory instance in StoreFileMetaData with one that wraps all I/O > requests with one of my wrappers is problematic as the tests never seem to > create an empty index, wrapping the I/O classes in Store.StoreDirectory > doesn't seem to work either as it keeps creating I/O classes outside this > point, causing it to read encrypted data without decrypting it. > > So I'm looking for the best place to hook this into elasticsearch. I've > verified that my encryption/decryption wrappers work with Lucene, so I'd > like to build a POC to make this work within elasticsearch. Any great ideas > from the community? πŸ™‚ > > Kind regards, > > Mattijs -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Hendrik](https://avatars.discourse-cdn.com/v4/letter/h/839c29/32.png) [@Hendrik](https://discuss.elastic.co/u/Hendrik)\ **Post date:** [November 6, 2013, 7:42am UTC](https://discuss.elastic.co/t/enabling-encrypted-indexes/12191/3 "2013-11-06T07:42:10Z")
i'am also interested in this Am Mittwoch, 6. November 2013 04:37:52 UTC+1 schrieb Mike Powers: > Hi Mattijs, > Did you ever come up with a solution for encrypting ES indexes? - It > seems that similar questions have been asked a few times, but I haven't > found anyone who has developed a solution? > > Thanks! > -Mike > > On Thursday, May 30, 2013 8:39:18 AM UTC-4, Mattijs Ugen wrote: > > > Friends, > > > > I've been trying to get elasticsearch to store indexes in an encrypted > > format. I know people have opted it here, but using full disk encryption is > > not a solution for me. The only place where the information that is being > > indexed is allowed to be available in plain form is in memory. This allows > > elasticsearch/Lucene to do its thing while the actual information cannot be > > read from disk without the proper key material. As far as I can find, > > there's no such implementation for elasticsearch as of yet. > > > > What I've done so far, is create I/O wrappers for Lucene 4.3 that accept > > read/seek/write requests like regular IndexInput and IndexOutput instances, > > but use CipherInputStream and CipherOutputStream to delegate the byte > > operations to a 'real' IndexInput or IndexOutput. This seems to work fine. > > I've found that elasticsearch does a comparable trick with wrapping index > > I/O instances, so I figured my code should be relatively easy to apply to > > the elasticsearch code base. > > > > Alas, this turns out to be untrue, as finding where to insert my wrappers > > on top of / below elasticsearch's is proving problematic. Wrapping the > > entire directory instance in StoreFileMetaData with one that wraps all I/O > > requests with one of my wrappers is problematic as the tests never seem to > > create an empty index, wrapping the I/O classes in Store.StoreDirectory > > doesn't seem to work either as it keeps creating I/O classes outside this > > point, causing it to read encrypted data without decrypting it. > > > > So I'm looking for the best place to hook this into elasticsearch. I've > > verified that my encryption/decryption wrappers work with Lucene, so I'd > > like to build a POC to make this work within elasticsearch. Any great ideas > > from the community? πŸ™‚ > > > > Kind regards, > > > > Mattijs -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 6, 2013, 2:33pm UTC](https://discuss.elastic.co/t/enabling-encrypted-indexes/12191/4 "2013-11-06T14:33:18Z")
Please be aware that ES also transmits data unencrypted over the wire between nodes. There is not much sense in encrypting data to disk, when the data is readable when listening to the network interface. Also a secure location of the keys is required which is a challenge in a distributed environment. The ES file I/O activity is scattered over many places. Lucene uses IndexReader/IndexWriter, but there is no ES I/O layer or something for the ES metadata add-ons. This would have to be refactored into the code. After an ES I/O refactoring, an encrypted ES FileChannel approach with a remote key store provider could lead in the right direction, like in Flume [https://issues.apache.org/jira/browse/FLUME-1424](https://issues.apache.org/jira/browse/FLUME-1424) In my personal opinion, encrypting every piece of data within an ES cluster is way too expensive, the result will be dog slow and ES won't deliver satisfactory results under these kind of requirements. And there are many other attack vectors if unauthorized access to ES nodes is possible. So, the safest place is to lock ES servers in a private network, and configure the servers so that nobody except authorized staff can access them. Security is not just about encrypting data, but about preventing intrusion into systems. JΓΆrg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:08am UTC](https://discuss.elastic.co/t/enabling-encrypted-indexes/12191/5 "2017-07-06T02:08:33Z")