I tried to enable var.convert_timezone: true
in {conf_path}/module.d/system
as I said in the question. And I thought it works.
Then I filtered the data from filebeat and found that some logs are in correct timestamp, some are not.
The logs in apache, mysql, and elastic server have a correct @timestamp
value. While the logs in syslog, elasticsearch.log, es_deprecation and es_gc are incorrect.
I've also checked my _ingest/pipeline and I don't think there's any problem in the syslog-pipeline.
"filebeat-7.2.0-system-syslog-pipeline" : {
"processors" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"""%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\[%{POSINT:process.pid:long}\])?: %{GREEDYMULTILINE:system.syslog.message}""",
"%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}",
"""%{TIMESTAMP_ISO8601:system.syslog.timestamp} %{SYSLOGHOST:host.hostname} %{DATA:process.name}(?:\[%{POSINT:process.pid:long}\])?: %{GREEDYMULTILINE:system.syslog.message}"""
],
"pattern_definitions" : {
"GREEDYMULTILINE" : "(.|\n)*"
},
"ignore_missing" : true
}
},
{
"remove" : {
"field" : "message"
}
},
{
"rename" : {
"field" : "system.syslog.message",
"target_field" : "message",
"ignore_missing" : true
}
},
{
"date" : {
"field" : "system.syslog.timestamp",
"target_field" : "@timestamp",
"formats" : [
"MMM d HH:mm:ss",
"MMM dd HH:mm:ss",
"ISO8601"
],
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "system.syslog.timestamp"
}
}
],
"on_failure" : [
{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}
],
"description" : "Pipeline for parsing Syslog messages."
}