Need help to setup Data curation layer for Elastic stack

i am trying to setup a curation layer for my current existing elastic cloud cluster, but am not sure how to implement this, could nod find much help over the web. can someone help ?

with curation layer, we are trying to achieve the below 4 enhancements.
please refer the details on terms for reference.

Data Curation:
There will be scenarios where curation is required prior to storing data in Elastic. Operations below cover the potential pre-processing. A curation layer is required to be built that would handle the operations given below.

Data Anonymisation - The content of the log messages as produced by the source is sensitive and should be obfuscated before storing into Elastic.

Data Enrichment and mapping - Log data to be enriched with information pulled from external sources (db, dns server, http url etc) prior to ingestion. For example: retrieve service information from a database, geographical information about an ip. The enrichment can also be done by adding labels. For example, utilise label field in ECS to tag an event as sensitive and then leverage roles to allow access at field and document level.
Data mapping - Mapping the contents of the log record from proprietary format to a standard like ECS.

Data Filtering - To reduce noise apply filters to filter out unwanted logs and metrics.

Routing - Route log messages to one or more destinations. For example to Elastic and Azure Sentinel.

Everything you listed can be done with Logstash.

You would need to send your data to Logstash, create the filter to perform the actions you need and then send your data to your cluster.

How you will do that depends entirely on how you collect the data, what is the format of your data, what you need to change etc, you will need to provide more information about your use case.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.