I THOUGHT THE PROBLEM HAS BEEN SOLVED, BUT IS'T NOT!
###########
Original Question:
I'm using filebeat to harvest logs directly to ES.
The timezone on my server is UTC +08:00 (Asia/Shanghai).
I used filebeat modules enable system elasticsearch kibana to configure filebeat to ingest Elasticsearch logs.
Everything worked fine, except a wierd problem: the kibana incorrectly displayed the timestamp. It moves 8 hours forward into the future.
Picture below shows the problem I met: the red line is the current time, and the '@timestamp' of every single syslog has been displayed 8 hours forward.
I read the JSON document and I found that the @timestamp is stored in UTC+8, which should be stored in UTC.
It's so weird!
The logs in apache, mysql, and elastic server have a correct @timestamp value. While the logs in system, elasticsearch are incorrect. For now.
This is the JSON struct of a single document:
{
"_index": "filebeat-7.2.0-2019.07.28-000001",
"_type": "_doc",
"_id": "qRLNPWwBaHGrHZ27tQUU",
"_version": 1,
"_score": null,
"_source": {
"agent": {
"hostname": "bigiron",
"id": "3eb06835-b68e-4b25-9ec1-1ebf0bac83a3",
"ephemeral_id": "fd8648df-4eab-4fd2-b915-97fb631211b6",
"type": "filebeat",
"version": "7.2.0"
},
"process": {
"name": "filebeat",
"pid": 28369
},
"log": {
"file": {
"path": "/var/log/syslog"
},
"offset": 1964308
},
"fileset": {
"name": "syslog"
},
"message": "2019-07-29T20:56:14.994+0800#011INFO#011log/harvester.go:253#011Harvester started for file: /var/log/apache2/access.log",
"cloud": {
"availability_zone": "cn-beijing-a",
"instance": {
"id": "i-2ze74ci7j8m68giheosd"
},
"provider": "ecs",
"region": "cn-beijing"
},
"input": {
"type": "log"
},
"@timestamp": "2019-07-29T20:56:14.000Z",
"system": {
"syslog": {}
},
"ecs": {
"version": "1.0.0"
},
"service": {
"type": "system"
},
"host": {
"hostname": "bigiron",
"os": {
"kernel": "4.4.0-146-generic",
"codename": "xenial",
"name": "Ubuntu",
"family": "debian",
"version": "16.04.6 LTS (Xenial Xerus)",
"platform": "ubuntu"
},
"containerized": false,
"name": "bigiron",
"id": "a76d0b1176a3140a8e07cb725964722b",
"architecture": "x86_64"
},
"event": {
"timezone": "+08:00",
"module": "system",
"dataset": "system.syslog"
}
},
"fields": {
"@timestamp": [
"2019-07-29T20:56:14.000Z"
],
"suricata.eve.timestamp": [
"2019-07-29T20:56:14.000Z"
]
},
"sort": [
1564433774000
]
}
###########
Update:
Filebeat cannot recognize the timezone from the log file. It assumes the time in syslog was stored in UTC, which was actually stored in UTC+8.
I tried to enable var.convert_timezone: true in {conf_path}/module.d/system and re-created the pipeline, Still not work.
My server time
UTC+8 2019/07/29 20:56:14
The time in syslog
Jul 29 20:56:14
The @timestamp from filebeat
2019-07-29T20:56:14.000Z
The @timestamp from filebeat should be
2019-07-29T12:56:14.000Z
system.yml:
- module: system
# Syslog
syslog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths: ["/var/log/syslog*"]
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
# Authorization logs
auth:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
elasticsearch.yml
- module: elasticsearch
# Server log
server:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
gc:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
audit:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
slowlog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
deprecation:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
I haven't yet.