How to parse @timestamp from Cloudfront logs?

[nhnicwaller (Nic Waller)]

I'm trying to use Filebeat to ship my logs from AWS CloudFront, but I can't figure out how to get the @timestamp field populated correctly. The log format is TSV (tab separated values) and the first two fields are date and time, like this:

2020-05-16	14:47:38	IAD89-C3	490	192.168.0.1	...

I'm using dissect to separate the fields and this works fine:

  - dissect:
      tokenizer: "%{timestamp.date}	%{timestamp.time}	%{edge_location}	%{response_bytes}	%{clientip}	%{method}	%{distribution}	%{uri_path}	%{response}	%{referer}	%{user_agent}	%{query}	%{cookie}	%{result_type}	%{request_id}	%{host_header}	%{protocol}	%{request_bytes}	%{duration}	%{x_forwarded_for}	%{ssl_protocol}	%{ssl_cipher}	%{edge_response_result_type}	%{cs_protocol_version}	%{fle_status}	%{fle_encrypted_fields}	%{c_port}	%{time_to_first_byte}	%{x_edge_detail_result_type}	%{sc_content_type}	%{sc_content_len}	%{sc_range_start}	%{sc_range_end}"
      field: "message"
      target_prefix: "aws.cloudfront"

But the problem is that @timestamp is not populated, and I cannot use the timestamp processor because it requires the entire timestamp to be in one field, and I can't figure out how to concatenate fields in filebeat.

I understand it's possible to change the output format used by CloudFront but I would rather not do that, because these logs are also part of a data lake so they should have a consistent format past and future.

[nhnicwaller (Nic Waller)]

I realized I can use the script processor to concatenate fields together. Here's how I'm populating @timestamp from CloudFront logs now.

config.yml

processors:
  - script:
      lang: javascript
      id: cloudfront_concatenate_timestamps
      file: ${path.config}/cloudfront_concatenate_timestamps.js
  - timestamp:
      field: "aws.cloudfront.datetime"
      layouts:
        - '2006-01-02T15:04:05Z'

cloudfront_concatenate_timestamp.js

function process(event) {
  event.Put('aws.cloudfront.datetime', event.Get('aws.cloudfront.date') + 'T' + event.Get('aws.cloudfront.time') + 'Z');
}