Ingesting America/Chicago timezone logs with no tz indicator

Elastic Stack Beats
1

Hi.

I have the following log entries. The timestamp is local system time but not UTC. I currently do not have a way to adjust the logs to include a tz.

D 2024-08-27T08:29:36,537 [Library] Logger - [MESSAGE] (1028445676) Received logging message
D 2024-08-27T08:29:37,307 [Library] Logger - [MESSAGE] Sent logging message
D 2024-08-27T08:29:39,537 [Library] Logger - [MESSAGE] (1028445676) Received logging message
D 2024-08-27T08:29:39,707 [Library] Logger - [MESSAGE] Sent logging message
D 2024-08-27T08:29:42,108 [Library] Logger - [MESSAGE] Sent logging message
D 2024-08-27T08:29:42,537 [Library] Logger - [MESSAGE] (1028445676) logging message
D 2024-08-27T08:29:44,508 [Library] Logger - [MESSAGE] Sent logging message
D 2024-08-27T08:29:45,537 [Library] Logger - [MESSAGE] (1028445676) logging message
D 2024-08-27T08:29:46,908 [Library] Logger - [MESSAGE] Sent logging message
D 2024-08-27T08:29:48,537 [Library] Logger - [MESSAGE] (1028445676) Received logging message

The issue I am having is that filebeat is ingesting these logs and they are stored in Elasticsearch as UTC. This causes confusion as Kibana is doing the tz conversion back to America/Chicago resulting in the time being off.

I added -timestamp but in Kibana I do not see the test field. This is assuming I am thinking about this the right way.

Any help will be appreciated.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/ingest_data/server*.log
  multiline:
    pattern: '^\t'
    match: after
  processors:
    - add_tags:
        tags: ['server']
    - dissect:
        field: message
        tokenizer: "%{level} %{timestamp} [%{thread}] %{logger} - %{msg}"
    - dissect:
        field: log.file.path
        tokenizer: "%{?path}/server.%{instance}.%{?ext}"
    - timestamp:
        field: "[dissect][timestamp]"
        timezone: Local
        layouts:
          - '2024-08-27T08:29:39,707'
          - '2024-08-27T08:29:48,537'
        target_field: test-field
2

What is the machine timezone?

I would explicitly set the timezone instead of using Local.

Have you tried using timezone: America/Chicago ?

3

The machine timezone is America/Chicago. The test-field doesn't get created and I see _dateparsefailure.

4

I have solved this by adding the add_locale to the filebeat processor.

    - add_locale:
        format: offset

Then I added this to the logstash filter. The field log-local-timezone now shows the expected time as compared to [dissect][timestamp]

    if [event][timezone] {
      date {
        match => ["[dissect][timestamp]", "yyyy-MM-dd'T'HH:mm:ss,SSS"]
        timezone => "%{[event][timezone]}"
        target => "log-local-timezone"
        add_tag => ["log-local-timezone"]
      }
    }
  }