Encryption In Elasticsearch

Elastic Stack Elasticsearch
1

HI Team,

We need to implement Encryption In Elasticsearch. As we know there are two types of encryption.

  1. Encryption in Transit.
  2. Encryption while data at REST.

Please help us how to achieve the same.

Thanks,
Debasis

2

Encryption in Transit is done by configuring Elasticsearch to use TLS on both the transport and http endpoints.

The documentation for it starts here and there are also plenty of posts already on the forum about it.

Encryption at REST is not done by Elasticsearch, but by the operating system as explained in this post, for linux normally you just enable dm-crypt on your server.

1 Like
3

Thanks @leandrojmp for response. Since encryption at REST depends on third party tool (OS) so is it avliable if we use basic version of Elasticsearch.

Thanks,
Debasis

4

@leandrojmp As further discussion with DEV team get to know that, they want to encrypt the specific fields in Elasticsearch not the entire record which received from the customer.

5

This needs to be done during ingestion, you can do it in both Logstash or using an Ingest pipeline.

Logstash has a fingerprint filter that can generate a hash of some string and Elasticsearch has a fingerprint processor that does the same thing.

But keep in mind thaty ou cannot search for the value befure you execute the fingerprint, only by the hash created.

For example, if you have this:

fieldName: unencrypted value, after you run the fingerprint filter or processor you can have fieldName: random-hash.

You cannot search fieldName for unencrypted value, but you can search using random-hash.

6

Thanks @leandrojmp for response. As you mentioned user need to search on random-hash value but how the end user or application will know the hash value of the actual value which is send by the customer.

7

The application will need to hash the search term using the same algorithm used when it was indexed. Naturally this will only work for exact matches, although you can make it case insensitive by lowercasing before both indexing and querying

8

You would need to do something like Christian mentioned, you will need to create the has of the entire value and search for it, it only works for exact matches.

For example, if you have a field named sensitive_message that needs to be encrypted and this field has this value The quick brown fox jumps over the lazy dog, you will create a hash of the entire string, and will only be able to search by the hash.

As mentioned:

But keep in mind that you cannot search for the value before you execute the fingerprint, only by the hash created.

Which I mean is, if you generated a hash for a field, you cannot search on the uncrypted value, only the hash.

1 Like
9

@leandrojmp Could you please share any blog reference, how to use fingerprint processor in filebeat.
I tried search in google did not find any blog reference how to use in file beat.

Thanks,
Debasis

10

I do not know any blog post about it.

But have you checked the documentation here?

You also have a fingerprint processor that can be used in an ingest pipeline in Elasticsearch, it is pretty similar and the documentation is here.

11

Although the fingerprint processor will replace fields with hashed values, remember that you will need to be able to exactly replicate this hash calculation in your application code if you want to be able to search of any of these hash values.