[Still Not Solved!] Filebeat cannot recognize timezone in syslog

Elastic Stack Beats
22

I have deleted filebeat-7.2.0-system-syslog-pipeline and filebeat-7.2.0-system-auth-pipeline

Then I ran filebeat using the configuration like this:

/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat -E filebeat.overwrite_pipelines=true

Then I use GET /_ingest/pipeline?pretty, but I couldn't find any new pipeline created.

The whole pipeline list after restarting the filebeat is here.

23

well... my apology, the new pipeline has been created, and it has a timezone property!
I looked further and found that only these pipelines have the timezone property:

filebeat-7.2.0-system-syslog-pipeline (recreaeted)
filebeat-7.2.0-system-auth-pipeline (recreated)
filebeat-7.2.0-elasticsearch-server-pipeline
filebeat-7.2.0-elasticsearch-slowlog-pipeline
filebeat-7.2.0-elasticsearch-audit-pipeline

Is that normal?

Unluckily, the @timestamp in syslogs are still incorrect... The timezone configuration in syslog now is like:

"timezone" : "{{ event.timezone }}"
24

There's other developers facing the same problems as mine! I hope you could figure out the problem and fix it. It will be so helpful.

25

Hello,

I have the same problem on Filebeat's elasticsearch module but it work's fine for the system module.
Let me share what i have done to fix it. However, my ELK stack is on development and not for production, hence following my procedure might cause loss of data.

My server timezone is in UTC +8:00 (Singapore)
Enabling var.convert_timezone: true converts my server timezone into UTC, which means - 8 hours.
On Kibana, Timezone for date formatting was left as default where it reads the timezone of the browser and +8 hours back from UTC.
Hence, showing the correct timing.

Re-indexing and Recreating Pipeline

  1. Stop Filebeat on all instance that utilize the old pipeline

  2. $ systemctl stop filebeat

  3. Enable UTC time conversion in system.yml

  4. Delete pipelines

  5. $ curl -XDELETE 'http://esnode1:9200/_ingest/pipeline/filebeat-*'

  6. Delete Related Indicies @ Index Management

  7. Since the logs shown in kibana are in wrong time, we should reindex all the logs

  8. In kibana, navigate to management > index management > select the index that stores the server logs

  9. Delete it

  10. Recreate the index

  11. $ filebeat setup -e

  12. $ filebeat setup --pipelines -modules="system"

  13. Restart filebeat

  14. $ systemctl start filebeat

  15. Observer logs tab on kibana and it should show logs with the correct timing.

26

Alright! I'll have a try. I have the problem on system as well as elasticsearch LOL.

27

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.