[Still Not Solved!] Filebeat cannot recognize timezone in syslog

[cosloli (Yang Juecheng)]

I have deleted filebeat-7.2.0-system-syslog-pipeline and filebeat-7.2.0-system-auth-pipeline

Then I ran filebeat using the configuration like this:

/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat -E filebeat.overwrite_pipelines=true

Then I use GET /_ingest/pipeline?pretty, but I couldn't find any new pipeline created.

The whole pipeline list after restarting the filebeat is here.

[cosloli (Yang Juecheng)]

well... my apology, the new pipeline has been created, and it has a timezone property!
I looked further and found that only these pipelines have the timezone property:

filebeat-7.2.0-system-syslog-pipeline (recreaeted)
filebeat-7.2.0-system-auth-pipeline (recreated)
filebeat-7.2.0-elasticsearch-server-pipeline
filebeat-7.2.0-elasticsearch-slowlog-pipeline
filebeat-7.2.0-elasticsearch-audit-pipeline

Is that normal?

Unluckily, the @timestamp in syslogs are still incorrect... The timezone configuration in syslog now is like:

"timezone" : "{{ event.timezone }}"

[cosloli (Yang Juecheng)]

There's other developers facing the same problems as mine! I hope you could figure out the problem and fix it. It will be so helpful.

@TsuWeiQuan

Hi,

I am still unable to solve this issue. Hence, i had to turn off the elasticsearch module due to this bug. It seems like the var.convert_timezone does not work well where it converts the timezone into UTC to store into elasticsearch.

[TsuWeiQuan]

Hello,

I have the same problem on Filebeat's elasticsearch module but it work's fine for the system module.
Let me share what i have done to fix it. However, my ELK stack is on development and not for production, hence following my procedure might cause loss of data.

My server timezone is in UTC +8:00 (Singapore)
Enabling var.convert_timezone: true converts my server timezone into UTC, which means - 8 hours.
On Kibana, Timezone for date formatting was left as default where it reads the timezone of the browser and +8 hours back from UTC.
Hence, showing the correct timing.

Re-indexing and Recreating Pipeline

1. Stop Filebeat on all instance that utilize the old pipeline

2. $ systemctl stop filebeat

3. Enable UTC time conversion in system.yml

4. Delete pipelines

5. $ curl -XDELETE 'http://esnode1:9200/_ingest/pipeline/filebeat-*'

6. Delete Related Indicies @ Index Management

7. Since the logs shown in kibana are in wrong time, we should reindex all the logs

8. In kibana, navigate to management > index management > select the index that stores the server logs

9. Delete it

10. Recreate the index

11. $ filebeat setup -e

12. $ filebeat setup --pipelines -modules="system"

13. Restart filebeat

14. $ systemctl start filebeat

15. Observer logs tab on kibana and it should show logs with the correct timing.

[cosloli (Yang Juecheng)]

Alright! I'll have a try. I have the problem on system as well as elasticsearch LOL.