# Challenges with parsing AWS EKS Logs - Custom Cloudwatch Integration **URL:** \ **Category:** Logs\ **Created:** [February 24, 2025, 8:27pm UTC](https://discuss.elastic.co/t/challenges-with-parsing-aws-eks-logs-custom-cloudwatch-integration/374964 "2025-02-24T20:27:25Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![BKgingersnap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkgingersnap/32/140989_2.png) [@BKgingersnap](https://discuss.elastic.co/u/BKgingersnap)\ **Post date:** [February 24, 2025, 8:27pm UTC](https://discuss.elastic.co/t/challenges-with-parsing-aws-eks-logs-custom-cloudwatch-integration/374964/1 "2025-02-24T20:27:25Z")
I'm working on parsing AWS EKS logs that are ingested into Elastic via custom Cloud watch integration + filebeat. The problem though is that right now, the entire Kubernetes audit log is captured in the message field and it contains complex deeply nested fields, and I don't think I have the knowledge / skill at the point to write a reliable processor myself. I was wondering if anyone had any ideas or strategies to do deal with this situation or if there were any special purpose tools or existing processors / pipelines that may be able to help. Thanks!!! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 24, 2025, 8:40pm UTC](https://discuss.elastic.co/t/challenges-with-parsing-aws-eks-logs-custom-cloudwatch-integration/374964/2 "2025-02-24T20:40:33Z")
Hi @BKgingersnap Welcome to the community. Always include the version you are on please. Perhaps take a look at this. > **[Bringing Your Cloud-Managed Kubernetes Audit Logs into Elasticsearch —...](https://www.elastic.co/observability-labs/blog/bringing-your-cloud-managed-kubernetes-audit-logs-into-elasticsearch)** > > How to bring your Cloud-Managed Kubernetes Audit Logs into Elasticsearch ---
**Author:** ![BKgingersnap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkgingersnap/32/140989_2.png) [@BKgingersnap](https://discuss.elastic.co/u/BKgingersnap)\ **Post date:** [February 24, 2025, 8:58pm UTC](https://discuss.elastic.co/t/challenges-with-parsing-aws-eks-logs-custom-cloudwatch-integration/374964/3 "2025-02-24T20:58:05Z")
Agent Version is 8.16.1 Ingest PIpelines: logs-aws.cloudwatch\_logs-2.24.0 logs-aws.cloudwatch\_logs@custom Mappings: logs-aws.cloudwatch\_logs@package Integration Policy: aws-usw2-cloudwatch-eks-prod-2 v2.24.0 Thanks for the article, will read through and see if this helps. ---
**Author:** ![BKgingersnap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkgingersnap/32/140989_2.png) [@BKgingersnap](https://discuss.elastic.co/u/BKgingersnap)\ **Post date:** [March 7, 2025, 11:22pm UTC](https://discuss.elastic.co/t/challenges-with-parsing-aws-eks-logs-custom-cloudwatch-integration/374964/4 "2025-03-07T23:22:17Z")
So I was able to follow the steps here: [Bringing Your Cloud-Managed Kubernetes Audit Logs into Elasticsearch — Elastic Observability Labs](https://www.elastic.co/observability-labs/blog/bringing-your-cloud-managed-kubernetes-audit-logs-into-elasticsearch) To parse EKS logs delivered via Cloudwatch to Kubernetes index Now I need to do the same thing with Azure Kubernetes Logs Currently AKS logs are delivered via filebeat on data\_stream.dataset: azure.activitylogs The AKS logs are broadly the same as the EKS logs except on the azure.activitylogs data stream the field containing the K8s audit logs is a flattened field vs a text field. So when I try to use the steps in the article I get an error. What would be the approach to deal with this? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 8, 2025, 3:36am UTC](https://discuss.elastic.co/t/challenges-with-parsing-aws-eks-logs-custom-cloudwatch-integration/374964/5 "2025-03-08T03:36:21Z")
> [@BKgingersnap](#): > > The AKS logs are broadly the same as the EKS logs except on the azure.activitylogs data stream the field containing the K8s audit logs is a flattened field vs a text field. How does the log looks like? Can you share some example? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 8, 2025, 3:58am UTC](https://discuss.elastic.co/t/challenges-with-parsing-aws-eks-logs-custom-cloudwatch-integration/374964/6 "2025-03-08T03:58:14Z")
As @leandrojmp said you are going to need to share detailed specifics ... "Broadly" matches is not going to get you there 🙂 And How exactly are you shipping the AKS Audit logs? Through Azure Diagnostics and Event hub? Please share your ingestion path and share you filebeat configuration... ---
**Author:** ![BKgingersnap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkgingersnap/32/140989_2.png) [@BKgingersnap](https://discuss.elastic.co/u/BKgingersnap)\ **Post date:** [March 10, 2025, 3:37pm UTC](https://discuss.elastic.co/t/challenges-with-parsing-aws-eks-logs-custom-cloudwatch-integration/374964/7 "2025-03-10T15:37:00Z")
Here is an example of the difference EKS Message log - Text / string data type with all logs contained with Json block . Parsed easily using Kubernetes default mapping ```auto {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"","stage":"ResponseComplete","requestURI":"","verb":"get","user":{"username":,"uid":"","groups":["system:serviceaccounts","system:serviceaccounts:default","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":[""],"authentication.kubernetes.io/pod-uid":[""]}},"sourceIPs":[""],"userAgent":"","objectRef":{"resource":"gateways","namespace":"","name":"","apiGroup":"","apiVersion":""},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-03-04T06:22:18.819232Z","stageTimestamp":"2025-03-04T06:22:18.822532Z","annotations":{"authorization.k8s.io/decision":"allow","" of ClusterRole \"" to ServiceAccount \"""}} ``` AKS azure.activitylogs.properties - Flattened field with "pod" and "containerID" fields before, Audit log Json block, then "stream" field after. ```auto { "pod": "", "containerID": "", "log": "{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\",\"auditID\":\"\",\"stage\":\"ResponseComplete\",\"requestURI\":\"",\"verb\":\"update\",\"user\":{\"username\":\"",\"groups\":[\"system:masters\",\"system:authenticated\"]},\"sourceIPs\":[\""],\"userAgent\":\"",\"objectRef\":{\"resource\":\"leases\",\"namespace\":\"kube-system\",\"name\":\"",\"uid\":\"",\"apiGroup\":\"coordination.k8s.io\",\"apiVersion\":\"v1\",\"resourceVersion\":\""},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2025-03-04T06:22:18.599549Z\",\"stageTimestamp\":\"2025-03-04T06:22:18.604251Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"\"}}\n", "stream": "" } ``` ---
**Author:** ![BKgingersnap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkgingersnap/32/140989_2.png) [@BKgingersnap](https://discuss.elastic.co/u/BKgingersnap)\ **Post date:** [March 13, 2025, 3:06pm UTC](https://discuss.elastic.co/t/challenges-with-parsing-aws-eks-logs-custom-cloudwatch-integration/374964/8 "2025-03-13T15:06:43Z")
I was able to map the AKS logs to the the Kubernetes default mapping by doing the following: Run the following pipeline which converts flattened field - "azure.activitylogs.properties" to text, run gsub to remove "{pod= to log=", run gsub to remove, ", stream=stdout, containerID=.\*", And then run the 2 pipelines in the article posted above (changing names as needed) ```auto [ { "convert": { "field": "azure.activitylogs.properties", "type": "string", "target_field": "message" } }, { "gsub": { "field": "message", "pattern": "\\{pod=.*?log=", "replacement": "" } }, { "gsub": { "field": "message", "pattern": ", stream=stdout, containerID=.*", "replacement": "" } } ] ```