Filebeat uses process time instead event time

Elastic Stack Beats
1

Hi all,

Hope you can help me out for the following.

We've setup and Elasticstack and configured filebeat to send json logs to elasticsearch, using this configuration:

    filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /log/job-json/job-json-*.log
  json.keys_under_root: true
  json.overwrite_keys: true
  json.expand_keys: false
  fields:
    - document_type: application_job_log
  fields_under_root: true
  keep_null: true
  publisher_pipeline.disable_host: true
  index: "app-job-%{+yyyy.MM.dd}"
  document_id: "logger"
- type: log
  enabled: true
  paths:
    - /log/error-json/error-json-*.log
  json.keys_under_root: true
  json.overwrite_keys: true
  json.expand_keys: false
  fields:
    document_type: application_error_log
  fields_under_root: true
  keep_null: true
  publisher_pipeline.disable_host: true
  index: "app-err-%{+yyyy.MM.dd}"
  document_id: "logger"

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.elasticsearch:
  enabled: true
  hosts: ["https://0.0.0.0:9200"]
  ssl.certificate_authorities: ["/some/pem/file.pem"]
  username: "ispy"
  password: "verysecret"
  indices:
    - index: "app-job-%{+yyyy.MM.dd}"
      when.contains:
        document_type: "application_job_log"
    - index: "app-err-%{+yyyy.MM.dd}"
      when.contains:
        document_type: "application_error_log"

above code outputs something like this:

    {
  "@timestamp": "2021-05-21T08:49:02.835Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.12.1",
    "raw_index": "app-job-2021.05.21"
  },
  "timestamp": "1621586880034",
  "level": "INFO",
  "logger": "com.ourcompany.beehive.core.internal.job.JobExecutor"
}

as you can see, there two timestamp fields:
@timestamp
and
timestamp

@timestamp = the time when filebeat reads the event
timestamp = the epoch time of the logged event

we want to use the logged event timestamp to be used as the @timestamp. this so that we have the events in a historical correct order.

However, we are unable to get it working.

Hope you guys and girls can help out in this matter.

Kr,
Nathan

2

You can use the filebeat timestamp processor, Timestamp | Filebeat Reference [7.12] | Elastic or the elasticsearch date ingest processor, Date processor | Elasticsearch Guide [7.12] | Elastic to parse the timestamp field into the @timestamp field.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.