Elasticsearch data encryption

Elastic Stack Elasticsearch
1

HI Everyone,

Currently i am using elastic 7.17.5 along with spring boot. Index creation and data save and retrieval is happening via Java only.

Below is my requirement,
I need to have a data saved in the Elasticsearch is should encrypted format. When some one directly curl the elastic they should see only the encrypted value not the actual value. In this case i need to do the searches(exact, Prefix, term, wild card and fuzzy) with scoring as well.

The same i need to if for the non text field like date, integer, Boolean etc as well. This case i need to support the range query option as well

Also i need to make the search to work as well.

Please suggest the approaches to achieve the above criteria.

2

Welcome back @Kesavan!

I assume you're using a self-managed Elasticsearch instance? You'll need to encrypt the data at OS level as it's not something Elasticsearch supports out of the box.

This article is a bit dated but it does recommend a few tools and considerations for applying encryption at REST.

Hope that helps!

3

Can you tell me a little about where this requirement comes from? e.g. is there a corporate policy at play here, if so can you quote what it actually says?

The requirement as you have described it:

is pretty unusual.

@carly.richmond covered the "data at rest" aspect, but as written your requirement is different.

4

@carly.richmond

We are running a Elasticsearch as a standalone in ubuntu VM and docker instance as well.

This case if the person breaks the dm-crypt saved elastic data is compromised right. I need a data level encryption as well.

it's a data at rest. Even dm-crypt customized the hacked person should see only the encrypted data not the actual plain text.

But my major requirement is i need to save the encrypted data without losing all Elasticsearch capabilities. Also same for not text fields as well.

5

Sorry @RainTown if i confused.

My requirement is encryption on data at rest .

6

I don't think you can do what you have in mind. If you need to search for data, the data must be readable by the Elasticsearch process so it's technically available in the memory and any user which could search for the data will be able to fetch back the data.

You can have fields which are only visible by admins, with field security, but it's not what you asked for I think.

dm-crypt is the way to go.

7

@dadoonet I am planning to go with 2 fields approach, one will have the encrypted data and another will have the plan text.

For example:
name - encrypted data

name_search-plain text.

Mapping i am doing below:
index: true
store: false
Excluded from _source

This case the name_search field is visible to curl command for all users?

This case is any way, some one can see what is saved in the name_search field?

8

What if someone runs a terms agg on your field?

9

which field ? name or name_search?

10

Your requirement seems to be drifting a bit, but interesting nonetheless.

PUT /yourindex
{
    "mappings": {
        "_source": {
            "enabled": false
        },
        "properties": {
            "name_search" : {
                "type" : "text",
                "index" : "true",
                "store" : "false"
             }
        }
    }
}

is indeed making it harder to retrieve the value of name_search field in each indexed doc, even if I had the "disk" on which this data is stored. As effectively only the analysed tokens, and which docs contain which tokens, are "known" from the data on the disk.

And if that disk is encrypted, I'd have to overcome that too.

But "harder" might not be enough, and I'd certainly not call it "secure" or "encrypted". Because access to the disk gives access to the Lucene data structures, which contain the tokens, and lucene tools could (theoretically) be used to rebuild the original data, at least to a close approximation. Note that I've never tried doing so, so I dont know how hard it would be.

11

It's not trivial, but it's much easier than bypassing dm-crypt in the first place :wink:

If you care about security enough to even ask this question then you definitely shouldn't be using an unsupported and unmaintained version.