# Preserve @timestamp Field for Custom Filebeat JSON logs

**URL:** <https://discuss.elastic.co/t/preserve-timestamp-field-for-custom-filebeat-json-logs/264495>\
**Category:** Beats\
**Tags:** fleet, filebeat\
**Created:** [February 16, 2021, 11:41pm UTC](https://discuss.elastic.co/t/preserve-timestamp-field-for-custom-filebeat-json-logs/264495 "2021-02-16T23:41:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tim\_Estes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tim_estes/32/64928_2.png) [@Tim\_Estes](https://discuss.elastic.co/u/Tim_Estes)\
**Post date:** [February 16, 2021, 11:41pm UTC](https://discuss.elastic.co/t/preserve-timestamp-field-for-custom-filebeat-json-logs/264495/1 "2021-02-16T23:41:54Z")

</div>

I've deployed agents across two machines that are ingesting custom log data stored in json files. However, the @timestamp field present in the log files are not getting preserved.

For example, here's a json log generated by my microservice (indentation added for readability):

```auto
{
    "@timestamp": "2021-02-16T22:53:58.294Z",
    "log.level": "info",
    "message": "Successfully uploaded file",
    "ecs": {"version": "1.6.0"},
    "log": {
        "logger": "app",
        "origin": {
            "file": {"line": 37, "name": "filing_operations.py"},
            "function": "staging_to_bucket",
        },
    }
}

```

But a few seconds later, and this message is turned into this document in my data stream (notice that the @timestamp field is different!!) Note: `log` and `cloud` fields have been redacted.

```auto
{
    "input" : {
      "type" : "log"
    },
    "agent" : {
      "hostname" : "file-staging-subscriber-deployment-fbc59c654-mgpnk",
      "name" : "file-staging-subscriber-deployment-fbc59c654-mgpnk",
      "id" : "06c61bd1-b71f-4d64-980c-4216985f8ec2",
      "type" : "filebeat",
      "ephemeral_id" : "179cb417-11cf-4660-bd82-0a2363a8b198",
      "version" : "7.10.2"
    },
    "@timestamp" : "2021-02-16T22:54:08.200Z",
    "ecs" : {
      "version" : "1.6.0"
    },
    "data_stream" : {
      "namespace" : "prod",
      "type" : "logs",
      "dataset" : "workflows"
    },
    "elastic_agent" : {
      "id" : "7037dfb8-5576-42e7-a22b-47f009991cca",
      "version" : "7.10.2",
      "snapshot" : false
    },
    "host" : {
      "hostname" : "file-staging-subscriber-deployment-fbc59c654-mgpnk",
      "os" : {
        "kernel" : "4.19.112+",
        "codename" : "stretch",
        "name" : "Debian GNU/Linux",
        "family" : "debian",
        "version" : "9 (stretch)",
        "platform" : "debian"
      },
      "containerized" : true,
      "ip" : [
        "10.44.7.43"
      ],
      "name" : "file-staging-subscriber-deployment-fbc59c654-mgpnk",
      "id" : "81ef01dec0f0eb6d6c0f3752b487b10e",
      "mac" : [
        "96:19:0d:d7:5a:7e"
      ],
      "architecture" : "x86_64"
    },
    "message" : "Successfully uploaded file to gs://briq-mark-tim/uploads/Microsoft/people/zapier_file.csv",
    "event" : {
      "dataset" : "workflows"
    }
}

```

_ **Main Question:** _  
Is there any way I can configure my filebeat to use the timestamp present on the json logs, and not the one given at index time?

I would like to use the original `2021-02-16T22:53:58.294Z` @timestamp value instead of `2021-02-16T22:54:08.200Z` that was assigned later.

Heres the elastic-agent.yaml I've been using to set up the agents:

```auto
id: 4b70f490-60d0-11eb-b745-f94eb598a8f6
revision: 3
outputs:
  default:
    type: elasticsearch
    hosts:
      - XXXXXX
    username: XXXXXX
    password: XXXXXX
agent:
  monitoring:
    enabled: false
    logs: false
    metrics: false
inputs:
  - id: 9f3cad30-60d0-11eb-b745-f94eb598a8f6
    name: workflows audit-trail logs
    revision: 2
    type: logfile
    use_output: default
    meta:
      package:
        name: log
        version: 0.4.6
    data_stream:
      namespace: prod
    streams:
      - id: logfile-log.log
        data_stream:
          dataset: workflows
        json.expand_keys: true
        json.keys_under_root: true
        json.add_error_key: true
        json.document_id: doc_id
        paths:
          - /Users/**/myService/myAuditLogs/*.log
          - /app/myAuditLogs/*.log

```

My microservices are on a managed kubernetes cloud cluster.

---

<div class="post-metadata">

**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\
**Post date:** [February 25, 2021, 9:54am UTC](https://discuss.elastic.co/t/preserve-timestamp-field-for-custom-filebeat-json-logs/264495/2 "2021-02-25T09:54:39Z")

</div>

maybe timestamp processor would help [Timestamp | Filebeat Reference [7.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 25, 2021, 11:54am UTC](https://discuss.elastic.co/t/preserve-timestamp-field-for-custom-filebeat-json-logs/264495/3 "2021-03-25T11:54:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.