# How to parse @timestamp from Cloudfront logs?

**URL:** <https://discuss.elastic.co/t/how-to-parse-timestamp-from-cloudfront-logs/233620>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 20, 2020, 9:21pm UTC](https://discuss.elastic.co/t/how-to-parse-timestamp-from-cloudfront-logs/233620 "2020-05-20T21:21:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nhnicwaller](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nhnicwaller/32/37645_2.png) [@nhnicwaller](https://discuss.elastic.co/u/nhnicwaller)\
**Post date:** [May 20, 2020, 9:21pm UTC](https://discuss.elastic.co/t/how-to-parse-timestamp-from-cloudfront-logs/233620/1 "2020-05-20T21:21:25Z")

</div>

I'm trying to use Filebeat to ship my logs from AWS CloudFront, but I can't figure out how to get the @timestamp field populated correctly. The [log format](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html#BasicDistributionFileFormat) is TSV (tab separated values) and the first two fields are date and time, like this:

```auto
2020-05-16	14:47:38	IAD89-C3	490	192.168.0.1	...

```

I'm using dissect to separate the fields and this works fine:

```auto
  - dissect:
      tokenizer: "%{timestamp.date}	%{timestamp.time}	%{edge_location}	%{response_bytes}	%{clientip}	%{method}	%{distribution}	%{uri_path}	%{response}	%{referer}	%{user_agent}	%{query}	%{cookie}	%{result_type}	%{request_id}	%{host_header}	%{protocol}	%{request_bytes}	%{duration}	%{x_forwarded_for}	%{ssl_protocol}	%{ssl_cipher}	%{edge_response_result_type}	%{cs_protocol_version}	%{fle_status}	%{fle_encrypted_fields}	%{c_port}	%{time_to_first_byte}	%{x_edge_detail_result_type}	%{sc_content_type}	%{sc_content_len}	%{sc_range_start}	%{sc_range_end}"
      field: "message"
      target_prefix: "aws.cloudfront"

```

But the problem is that @timestamp is not populated, and I cannot use the [timestamp processor](https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html) because it requires the entire timestamp to be in one field, and I can't figure out how to concatenate fields in filebeat.

I understand it's possible to change the output format used by CloudFront but I would rather not do that, because these logs are also part of a data lake so they should have a consistent format past and future.

---

<div class="post-metadata">

**Author:** ![nhnicwaller](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nhnicwaller/32/37645_2.png) [@nhnicwaller](https://discuss.elastic.co/u/nhnicwaller)\
**Post date:** [May 25, 2020, 6:37pm UTC](https://discuss.elastic.co/t/how-to-parse-timestamp-from-cloudfront-logs/233620/2 "2020-05-25T18:37:44Z")

</div>

I realized I can use the [script processor](https://www.elastic.co/guide/en/beats/filebeat/7.8/processor-script.html) to concatenate fields together. Here's how I'm populating `@timestamp` from CloudFront logs now.

config.yml

```auto
processors:
  - script:
      lang: javascript
      id: cloudfront_concatenate_timestamps
      file: ${path.config}/cloudfront_concatenate_timestamps.js
  - timestamp:
      field: "aws.cloudfront.datetime"
      layouts:
        - '2006-01-02T15:04:05Z'

```

cloudfront\_concatenate\_timestamp.js

```auto
function process(event) {
  event.Put('aws.cloudfront.datetime', event.Get('aws.cloudfront.date') + 'T' + event.Get('aws.cloudfront.time') + 'Z');
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2020, 6:37pm UTC](https://discuss.elastic.co/t/how-to-parse-timestamp-from-cloudfront-logs/233620/3 "2020-06-22T18:37:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.