Filebeat's elasticsearch module output wrong timestamp on Kibana Logs

[TsuWeiQuan]

Hello,

I am running 3 elasticsearch nodes and trying to output their logs using filebeat elasticsearch modules.
I was setting up filebeat logging for elasticsearch module and the logs for system.yml modules is in the correct timestamp. However, when i enable elasticsearch.yml module, the timestamp jumped forward. I have no idea why.

All my ELK machines are in SGT time where typing date returns me Wed May 29 16:24:05 +08 2019

This picture depicts the issue where the time jumped from 13:09 to 21:04:

I have enabled var.convert_timezone: true and also changed the timezone to Asia/Singapore in Kibana's advance settings.

This is my elasticsearch.yml file from Filebeat modules.d folder:

- module: elasticsearch
  # Server log
  server:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  gc:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  audit:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  slowlog:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  deprecation:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

Solutions tried but failed:

1. Stop filebeat service
2. Delete the index from this particular host that is sending the wrong timestamp logs
3. Delete the pipeline and re-initializing it
   a) curl -XDELETE 'http://elasticsearch:9200/_ingest/pipeline/filebeat-*'
   b) filebeat setup -e
   c) service filebeat start

I have also tried to add in "timezone" : "Asia/Singapore" in /usr/share/filebeat/module/elasticsearch/server/ingest/pipeline.json but it does not solve the problem

    "date": {
        "field": "elasticsearch.server.timestamp",
        "target_field": "@timestamp",
        "formats": [
            "ISO8601"
        ],
        "timezone" : "Asia/Singapore",
        {< if .convert_timezone >}"timezone": "{{ event.timezone }}",{< end >}
        "ignore_failure": true
    }
},

How can i solve this?