i am trying to setup a curation layer for my current existing elastic cloud cluster, but am not sure how to implement this, could nod find much help over the web. can someone help ?
with curation layer, we are trying to achieve the below 4 enhancements.
please refer the details on terms for reference.
Data Curation:
There will be scenarios where curation is required prior to storing data in Elastic. Operations below cover the potential pre-processing. A curation layer is required to be built that would handle the operations given below.
Data Anonymisation - The content of the log messages as produced by the source is sensitive and should be obfuscated before storing into Elastic.
Data Enrichment and mapping - Log data to be enriched with information pulled from external sources (db, dns server, http url etc) prior to ingestion. For example: retrieve service information from a database, geographical information about an ip. The enrichment can also be done by adding labels. For example, utilise label field in ECS to tag an event as sensitive and then leverage roles to allow access at field and document level.
Data mapping - Mapping the contents of the log record from proprietary format to a standard like ECS.
Data Filtering - To reduce noise apply filters to filter out unwanted logs and metrics.
Routing - Route log messages to one or more destinations. For example to Elastic and Azure Sentinel.