How to parse AWS Cloudfront logs

[leo-baltus]

Tring to get filebeat to parse AWS Cloudfront logs.
SQS seem to work. but once in ES I see:

Provided Grok expressions do not match field value: [2022-01-06\t20:06:28\tFRA56-C2\t1729\t3.125.241.170\tGET\tdtc81dn1qkg0w.cloudfront.net\t/.well-known/...

filebeat.yml:

cloud.id: "${ELASTICSEARCH_FILEBEAT_CLOUD_ID}"
cloud.auth: "${ELASTICSEARCH_FILEBEAT_CLOUD_AUTH}"
filebeat.modules:
- module: aws
  s3access:
    enabled: true
    var.queue_url: "${SQS_QUEUE}"
    var.access_key_id: "${AWS_ACCESS_KEY_ID}"
    var.secret_access_key: "${AWS_SECRET_ACCESS_KEY}"

Tried elb and aws-s3 to no avail.

I am running filebeat in k8s dockerimage: docker.elastic.co/beats/filebeat:7.16.2

Cloudfront logs are documented
here

Sample log:

#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end
2022-01-06	14:37:55	MUC50-P1	405	3.125.241.170	GET	dtc819989700w.cloudfront.net	/versioninfo/version	200	-	curl/7.68.0	-	-	Miss	aekvP2AgpYb1uYYV1dd8lCjcUwhEJGQucpuvgzafdJ1XXscToIlnPg==	xxxxxxxxxx.yyyyyyyyy.nl	https	61	0.051	-	TLSv1.3	TLS_AES_128_GCM_SHA256	Miss	HTTP/2.0	-	-	49716	0.051	Miss	text/plain;%20charset=utf-8	-	-	-

Any help or pointers are welcome.

[legoguy1000 (Alex)]

CloudFront logs and S3 access logs are not the same. You'll have to use a manualaws-s3 input and define a custom ingest pipeline to parse it.

[leo-baltus]

Thank you. Any pointers on examples or docs? there's not a great deal of information on this it seems.

[legoguy1000 (Alex)]

AWS S3 input | Filebeat Reference [7.16] | Elastic for how to setup the input. I built this pipeline last night, [AWS] Add CloudFront logs datastream by legoguy1000 · Pull Request #2476 · elastic/integrations · GitHub. You'll just have to point filebeat to this pipeline in ES either at the input or the out.

[leo-baltus]

I'm very impressed! Hope this PR gets accepted soon.

Thank you.