Encryption In Elasticsearch

Elastic Stack Elasticsearch
1

HI Team,

We need to implement Encryption In Elasticsearch. As we know there are two types of encryption.

  1. Encryption in Transit.
  2. Encryption while data at REST.

Please help us how to achieve the same.

Thanks,
Debasis

2

Encryption in Transit is done by configuring Elasticsearch to use TLS on both the transport and http endpoints.

The documentation for it starts here and there are also plenty of posts already on the forum about it.

Encryption at REST is not done by Elasticsearch, but by the operating system as explained in this post, for linux normally you just enable dm-crypt on your server.

1 Like
3

Thanks @leandrojmp for response. Since encryption at REST depends on third party tool (OS) so is it avliable if we use basic version of Elasticsearch.

Thanks,
Debasis

4

@leandrojmp As further discussion with DEV team get to know that, they want to encrypt the specific fields in Elasticsearch not the entire record which received from the customer.

5

This needs to be done during ingestion, you can do it in both Logstash or using an Ingest pipeline.

Logstash has a fingerprint filter that can generate a hash of some string and Elasticsearch has a fingerprint processor that does the same thing.

But keep in mind thaty ou cannot search for the value befure you execute the fingerprint, only by the hash created.

For example, if you have this:

fieldName: unencrypted value, after you run the fingerprint filter or processor you can have fieldName: random-hash.

You cannot search fieldName for unencrypted value, but you can search using random-hash.

6

Thanks @leandrojmp for response. As you mentioned user need to search on random-hash value but how the end user or application will know the hash value of the actual value which is send by the customer.

7

The application will need to hash the search term using the same algorithm used when it was indexed. Naturally this will only work for exact matches, although you can make it case insensitive by lowercasing before both indexing and querying

8

You would need to do something like Christian mentioned, you will need to create the has of the entire value and search for it, it only works for exact matches.

For example, if you have a field named sensitive_message that needs to be encrypted and this field has this value The quick brown fox jumps over the lazy dog, you will create a hash of the entire string, and will only be able to search by the hash.

As mentioned:

But keep in mind that you cannot search for the value before you execute the fingerprint, only by the hash created.

Which I mean is, if you generated a hash for a field, you cannot search on the uncrypted value, only the hash.

1 Like
9

@leandrojmp Could you please share any blog reference, how to use fingerprint processor in filebeat.
I tried search in google did not find any blog reference how to use in file beat.

Thanks,
Debasis

10

I do not know any blog post about it.

But have you checked the documentation here?

You also have a fingerprint processor that can be used in an ingest pipeline in Elasticsearch, it is pretty similar and the documentation is here.

11

Although the fingerprint processor will replace fields with hashed values, remember that you will need to be able to exactly replicate this hash calculation in your application code if you want to be able to search of any of these hash values.

12

@Christian_Dahlqvist and @leandrojmp As you mentioned fingerprint but it is not same as of encryption. Please correct me if I am wrong , fingerprint will generate a hash value but encryption is a different thing.

Thanks,
Debasis

13

Well, hashing can be seem as a one-way encryption, some places refer to it in this way.

But since encryption means that you can also decrypt the data, then yes, they are different things.

In this context it is all you have, so the fingerprint filter will work as a one-way process to encrypt your data, but you cannot decrypt it.

14

Thanks @leandrojmp for confirmation. Then how we can achieve data encryption at REST apart from dm-crypt which is a native tool to OS.

Thanks,
Debasis

15

I believe you need to handle this in your application/code. As far as I know there is no built in support within the stack.

16

There is nothing in the stack that would allow you to do that as already mentioned.

If you want to be able to encrypt and decrypt data stored in Elasticsearch, you need to do that on your application or code.