How to Handle Selective Masking and Reversible Encryption for PII in Elasticsearch Ingestion

Elastic Stack Elasticsearch
1

Hi Folks,

I’m working on a use case where we ingest API response payloads into the Elastic Stack (Logstash → Elasticsearch). The payload contains JSON data with a mix of important application-level information (like response codes/messages) and personally identifiable information (PII) such as:

  • idNumber
  • emailAddress
  • cellphoneNumber
  • surname
  • physicalAddrLine1

The payload comes in as a long string, e.g.:

BANKA formatted request with mapping: {<structured JSON with PII>}

Our Objective:

  • Retain important fields like responseCode/message
  • Mask or anonymize PII fields
  • For some fields (like idNumber), we may need reversible encryption instead of irreversible hashing using the fingerprint filter plugin, to allow re-identification under secure conditions

Please let me know if this is achievable with Elastic Stack.

Kind regards,
Souvik

2

Hi @Souvik_Das

Perhaps take a look at these 2 blogs to get some ideas

Structured / patterns are pretty easy to detect and Redact / mask...

Unstructured like address and Names may require the use of NLP.

With respect to field level 2 way encryption,
Elasticsearch does not natively support that.

You will need to build that at your application layer (yes that can be done with some work ) or perhaps use a 3rd party solution.

1 Like