Hello,
I am running 3 elasticsearch nodes and trying to output their logs using filebeat elasticsearch modules.
I was setting up filebeat logging for elasticsearch module and the logs for system.yml modules is in the correct timestamp. However, when i enable elasticsearch.yml module, the timestamp jumped forward. I have no idea why.
All my ELK machines are in SGT time where typing date
returns me Wed May 29 16:24:05 +08 2019
This picture depicts the issue where the time jumped from 13:09 to 21:04:
I have enabled var.convert_timezone: true
and also changed the timezone to Asia/Singapore
in Kibana's advance settings.
This is my elasticsearch.yml file from Filebeat modules.d folder:
- module: elasticsearch
# Server log
server:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
gc:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
audit:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
slowlog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
deprecation:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
Solutions tried but failed:
- Stop filebeat service
- Delete the index from this particular host that is sending the wrong timestamp logs
- Delete the pipeline and re-initializing it
a) curl -XDELETE 'http://elasticsearch:9200/_ingest/pipeline/filebeat-*'
b) filebeat setup -e
c) service filebeat start
I have also tried to add in "timezone" : "Asia/Singapore"
in /usr/share/filebeat/module/elasticsearch/server/ingest/pipeline.json
but it does not solve the problem
"date": {
"field": "elasticsearch.server.timestamp",
"target_field": "@timestamp",
"formats": [
"ISO8601"
],
"timezone" : "Asia/Singapore",
{< if .convert_timezone >}"timezone": "{{ event.timezone }}",{< end >}
"ignore_failure": true
}
},
How can i solve this?