I've deployed agents across two machines that are ingesting custom log data stored in json files. However, the @timestamp field present in the log files are not getting preserved.
For example, here's a json log generated by my microservice (indentation added for readability):
{
"@timestamp": "2021-02-16T22:53:58.294Z",
"log.level": "info",
"message": "Successfully uploaded file",
"ecs": {"version": "1.6.0"},
"log": {
"logger": "app",
"origin": {
"file": {"line": 37, "name": "filing_operations.py"},
"function": "staging_to_bucket",
},
}
}
But a few seconds later, and this message is turned into this document in my data stream (notice that the @timestamp field is different!!) Note: log and cloud fields have been redacted.
{
"input" : {
"type" : "log"
},
"agent" : {
"hostname" : "file-staging-subscriber-deployment-fbc59c654-mgpnk",
"name" : "file-staging-subscriber-deployment-fbc59c654-mgpnk",
"id" : "06c61bd1-b71f-4d64-980c-4216985f8ec2",
"type" : "filebeat",
"ephemeral_id" : "179cb417-11cf-4660-bd82-0a2363a8b198",
"version" : "7.10.2"
},
"@timestamp" : "2021-02-16T22:54:08.200Z",
"ecs" : {
"version" : "1.6.0"
},
"data_stream" : {
"namespace" : "prod",
"type" : "logs",
"dataset" : "workflows"
},
"elastic_agent" : {
"id" : "7037dfb8-5576-42e7-a22b-47f009991cca",
"version" : "7.10.2",
"snapshot" : false
},
"host" : {
"hostname" : "file-staging-subscriber-deployment-fbc59c654-mgpnk",
"os" : {
"kernel" : "4.19.112+",
"codename" : "stretch",
"name" : "Debian GNU/Linux",
"family" : "debian",
"version" : "9 (stretch)",
"platform" : "debian"
},
"containerized" : true,
"ip" : [
"10.44.7.43"
],
"name" : "file-staging-subscriber-deployment-fbc59c654-mgpnk",
"id" : "81ef01dec0f0eb6d6c0f3752b487b10e",
"mac" : [
"96:19:0d:d7:5a:7e"
],
"architecture" : "x86_64"
},
"message" : "Successfully uploaded file to gs://briq-mark-tim/uploads/Microsoft/people/zapier_file.csv",
"event" : {
"dataset" : "workflows"
}
}
Main Question:
Is there any way I can configure my filebeat to use the timestamp present on the json logs, and not the one given at index time?
I would like to use the original 2021-02-16T22:53:58.294Z @timestamp value instead of 2021-02-16T22:54:08.200Z that was assigned later.
Heres the elastic-agent.yaml I've been using to set up the agents:
id: 4b70f490-60d0-11eb-b745-f94eb598a8f6
revision: 3
outputs:
default:
type: elasticsearch
hosts:
- XXXXXX
username: XXXXXX
password: XXXXXX
agent:
monitoring:
enabled: false
logs: false
metrics: false
inputs:
- id: 9f3cad30-60d0-11eb-b745-f94eb598a8f6
name: workflows audit-trail logs
revision: 2
type: logfile
use_output: default
meta:
package:
name: log
version: 0.4.6
data_stream:
namespace: prod
streams:
- id: logfile-log.log
data_stream:
dataset: workflows
json.expand_keys: true
json.keys_under_root: true
json.add_error_key: true
json.document_id: doc_id
paths:
- /Users/**/myService/myAuditLogs/*.log
- /app/myAuditLogs/*.log
My microservices are on a managed kubernetes cloud cluster.