Filebeat-6.2.3： timestamp issue with output.kafka

Elastic Stack Beats
1

filebeat version: filebeat-6.2.3
kafka version: 0.10.2.1
filebeat.yml as follows:

filebeat.prospectors:

  • type: log
    enabled: ture
    paths:
    • /opt/filebeat/test.log
      fields:
      log_topic: test
      tag: bdu

output.kafka:
hosts: ["kafka1", "kafka2", "kafka3"]
topic: '%{[fields.log_topic]}'
required_ack: 1

It works perfect, but I have a requirement for changing the default @timestamp format (ISO 8601)
from: "@timestamp": "2018-04-12T10:58:01.787Z"
to: "@timestamp": "2018-04-12T18:58:01.787+08:00"

and I found in the /var/lib/filebeat/registry
the timestamp is exactly the format I need

is it possible to get the timestamp format in /var/lib/filebeat/registry ?

is there any other solution like getting the local time in fields ?

2

Have a look at https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html
date {
match => [ "@timestamp", "YYYY-MM-ddddTHH:mm:ssZZ" ]
}
or something like this could work.
You need some testing. Also i wouldn't quite recommend messing with timestamp itself, not sure how kibana/elastic might be affected.Try to create a second field "custom_date"(?) and test on that.

Edit: My bad you are not using logstash, you can't use this approach unless you add logstash in your stack.

3

Thanks for the respond

So is it possible to have a field like this:

fields:
timestamp: {local host time}

Cause I haven’t figured out what variable to deal with that