# 1 Hour Difference on showing Logs in ELK/Kibana

**URL:** <https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744>\
**Category:** Elasticsearch\
**Created:** [March 30, 2021, 7:56am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744 "2021-03-30T07:56:23Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![radu990](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@radu990](https://discuss.elastic.co/u/radu990)\
**Post date:** [March 30, 2021, 7:56am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/1 "2021-03-30T07:56:24Z")

</div>

Hello,

I have set a Filebeat agent on a server for Apache logs (using apache module), and observed that the logs are appearing on Kibana/ELK with 1 hour delay.  
Any idea what can be the problem? The time/timezone settings are correct on both ELK/Kibana and server in case. Checking the apache logs directly on the server displays ok all the logs, with the corect time. But ELK/Kibana will show them only 1 hour later.  
Observed that there is a difference in timestamp and event.created/event.ingested. Maybe it's related...

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/c/6c256f9f44b2145f4ef46102ec1650414c117276.png)

---

<div class="post-metadata">

**Author:** ![radu990](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@radu990](https://discuss.elastic.co/u/radu990)\
**Post date:** [April 2, 2021, 8:56am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/2 "2021-04-02T08:56:26Z")

</div>

Anybody? Any ideas why it happens so?  
I've compared with other servers where everything works well. Same timezone is used, same settings everywhere in filebeat.

Removed filebeat with manually removing all folders/files related to filebeat, and same delay happens after reinstalling and configuring from scratch.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 2, 2021, 9:05am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/3 "2021-04-02T09:05:06Z")

</div>

I can see that the `@timestamp` field does not seem to contain any TZ indication.  
But could you share a json document so we can check the exact values in the JSON document?

Did you see this? [Apache module | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-apache.html#_time_zone_support_2)

> This module parses logs that don’t contain time zone information. **For these logs, Filebeat reads the local time zone and uses it when parsing to convert the timestamp to UTC**. The time zone to be used for parsing is included in the event in the `event.timezone` field.
> 
> To disable this conversion, the `event.timezone` field can be removed with the `drop_fields` processor.
> 
> If logs are originated from systems or applications with a different time zone to the local one, the `event.timezone` field can be overwritten with the original time zone using the `add_fields` processor.

---

<div class="post-metadata">

**Author:** ![radu990](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@radu990](https://discuss.elastic.co/u/radu990)\
**Post date:** [April 2, 2021, 9:15am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/4 "2021-04-02T09:15:19Z")

</div>

Sure, please see below an example of a JSON document

```
{
  "_index": "filebeat-7.12.0-2021.03.30-000001",
  "_type": "_doc",
  "_id": "OT2akXgBrNnms5-qprv7",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "hostname": "domain.com",
      "name": "domain.com",
      "id": "1f0e7b95-79a1-41f1-9923-ddb2eee399d0",
      "type": "filebeat",
      "ephemeral_id": "ff5d0cdb-2875-4e65-95b2-3c2b98c96ba4",
      "version": "7.12.0"
    },
    "log": {
      "file": {
        "path": "/local/notesdata/weblogs/access04022021.log"
      },
      "offset": 4378390
    },
    "destination": {
      "domain": "domain.com"
    },
    "source": {
      "ip": "10.0.0.5"
    },
    "fileset": {
      "name": "access"
    },
    "url": {
      "original": "/ADB/revenues.nsf/icon11.gif"
    },
    "input": {
      "type": "log"
    },
    "apache": {
      "access": {}
    },
    "@timestamp": "2021-04-02T09:02:06.000Z",
    "ecs": {
      "version": "1.8.0"
    },
    "service": {
      "type": "apache"
    },
    "host": {
      "hostname": "domain.com",
      "os": {
        "kernel": "2.6.32-573.18.1.el6.x86_64",
        "codename": "Final",
        "name": "CentOS",
        "family": "redhat",
        "type": "linux",
        "version": "6.7 (Final)",
        "platform": "centos"
      },
      "containerized": false,
      "ip": [
        "192.168.1.60",
        "fe80::215:5dff:fe01:3301"
      ],
      "name": "domain.com",
      "id": "41d334d3044ca22a1daa6aae00000022",
      "mac": [
        "00:15:5d:01:33:01"
      ],
      "architecture": "x86_64"
    },
    "http": {
      "request": {
        "referrer": "https://domain.com/",
        "method": "GET"
      },
      "response": {
        "status_code": 200,
        "body": {
          "bytes": 1403
        }
      },
      "version": "1.1"
    },
    "event": {
      "ingested": "2021-04-02T08:02:14.846051490Z",
      "kind": "event",
      "created": "2021-04-02T08:02:13.801Z",
      "module": "apache",
      "category": "web",
      "dataset": "apache.access",
      "outcome": "success"
    },
    "user_agent": {
      "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36",
      "os": {
        "name": "Windows",
        "version": "10",
        "full": "Windows 10"
      },
      "name": "Chrome",
      "device": {
        "name": "Other"
      },
      "version": "89.0.4389.90"
    },
    "username": "User/O=Domain/C=CZ"
  },
  "fields": {
    "event.ingested": [
      "2021-04-02T08:02:14.846Z"
    ],
    "@timestamp": [
      "2021-04-02T09:02:06.000Z"
    ],
    "event.created": [
      "2021-04-02T08:02:13.801Z"
    ]
  },
  "highlight": {
    "suricata.eve.http.url": [
      "@kibana-highlighted-field@/ADB/revenues.nsf/icon11.gif@/kibana-highlighted-field@"
    ],
    "url.original": [
      "@kibana-highlighted-field@/ADB/revenues.nsf/icon11.gif@/kibana-highlighted-field@"
    ],
    "host.hostname": [
      "@kibana-highlighted-field@domain.com@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1617350534846
  ]
}
```

---

<div class="post-metadata">

**Author:** ![radu990](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@radu990](https://discuss.elastic.co/u/radu990)\
**Post date:** [April 2, 2021, 9:17am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/5 "2021-04-02T09:17:32Z")

</div>

My current Apache pipeline.yml . I don't see anything related to event.timezone there ☹

```
[root@server]# cat /usr/share/filebeat/module/apache/access/ingest/pi peline.yml
description: "Pipeline for parsing Apache HTTP Server access logs. Requires the geoip and user_agent plugins."

processors:
- set:
    field: event.ingested
    value: '{{_ingest.timestamp}}'
- grok:
    field: message
    patterns:
    - '%{IPORHOST:destination.domain} %{IPORHOST:source.ip} - %{DATA:user.name} \[%{HTTPDATE:apache.access.time}\]
      "(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.ve rsion}|-)?"
      %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.by tes:long}|-)(
      "%{DATA:http.request.referrer}")?( "%{DATA:user_agent.original}")?'
    - '%{IPORHOST:source.address} - %{DATA:user.name} \[%{HTTPDATE:apache.access .time}\]
      "(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.ve rsion}|-)?"
      %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.by tes:long}|-)(
      "%{DATA:http.request.referrer}")?( "%{DATA:user_agent.original}")?'
    - '%{IPORHOST:source.address} - %{DATA:user.name} \[%{HTTPDATE:apache.access .time}\]
      "-" %{NUMBER:http.response.status_code:long} -'
    - \[%{HTTPDATE:apache.access.time}\] %{IPORHOST:source.address} %{DATA:apach e.access.ssl.protocol}
      %{DATA:apache.access.ssl.cipher} "%{WORD:http.request.method} %{DATA:url.o riginal}
      HTTP/%{NUMBER:http.version}" (-|%{NUMBER:http.response.body.bytes:long})
    - '%{IPORHOST:source.ip} (%{IPORHOST:destination.domain}.*|.*) ("CN=%{DATA:u sername}"|"%{DATA:username}"|-) \[%{HTTPDATE:apache.access.time}\] "(?:%{WORD:ht tp.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-|%{GREEDYDA TA:url.original})?" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.re sponse.body.bytes:long}|-) ("%{DATA:http.request.referrer}") ("%{DATA:user_agent .original}")'
    - '%{IPORHOST:source.ip} (%{IPORHOST:destination.domain}.*|.*) - \[%{HTTPDAT E:apache.access.time}\] "(?:%{WORD:http.request.method} %{DATA:url.original} HTT P/%{NUMBER:http.version}|-|%{GREEDYDATA:url.original})?" %{NUMBER:http.response. status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-) ("%{DATA:http.re quest.referrer}") ("%{DATA:user_agent.original}") (?:%{NUMBER:aliator}|-) (\".*I NOTES_LOGIN_ID=%{DATA:username}|.*\")'
    ignore_missing: true
- remove:
    field: message
- set:
    field: event.kind
    value: event
- set:
    field: event.category
    value: web
- set:
    field: event.outcome
    value: success
    if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_co de < 400"
- set:
    field: event.outcome
    value: failure
    if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_co de > 399"
- grok:
    field: source.address
    ignore_missing: true
    patterns:
    - ^(%{IP:source.ip}|%{HOSTNAME:source.domain})$
- rename:
    field: '@timestamp'
    target_field: event.created
- date:
    field: apache.access.time
    target_field: '@timestamp'
    formats:
    - dd/MMM/yyyy:H:m:s Z
    ignore_failure: true
- remove:
    field: apache.access.time
    ignore_failure: true
- user_agent:
    field: user_agent.original
    ignore_failure: true
- geoip:
    field: source.ip
    target_field: source.geo
    ignore_missing: true
- geoip:
    database_file: GeoLite2-ASN.mmdb
    field: source.ip
    target_field: source.as
    properties:
    - asn
    - organization_name
    ignore_missing: true
- rename:
    field: source.as.asn
    target_field: source.as.number
    ignore_missing: true
- rename:
    field: source.as.organization_name
    target_field: source.as.organization.name
    ignore_missing: true
- set:
    field: tls.cipher
    value: '{{apache.access.ssl.cipher}}'
    ignore_empty_value: true

- script:
    lang: painless
    if: ctx?.apache?.access?.ssl?.protocol != null
    source: >-
      def parts = ctx.apache.access.ssl.protocol.toLowerCase().splitOnToken("v") ;
      if (parts.length != 2) {
        return;
      }
      if (parts[1].contains(".")) {
        ctx.tls.version = parts[1];
      } else {
        ctx.tls.version = parts[1] + ".0";
      }
      ctx.tls.version_protocol = parts[0];

on_failure:
- set:
    field: error.message
    value: '{{ _ingest.on_failure_message }}'
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 2, 2021, 9:34am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/6 "2021-04-02T09:34:10Z")

</div>

What is the exact Apache log line which correspond to this event?  
What was you local date when this event happened?  
What is your current timezone?

---

<div class="post-metadata">

**Author:** ![radu990](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@radu990](https://discuss.elastic.co/u/radu990)\
**Post date:** [April 2, 2021, 11:20am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/7 "2021-04-02T11:20:59Z")

</div>

Log entry

```auto
10.0.0.5 domain.com "CN=User/O=Company/C=CZ" [02/Apr/2021:10:02:06 +0100] "GET /ADB/revenues.nsf/icon11.gif HTTP/1.1" 200 1403 "https://domain.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" 139857020059654 "_ga=GA1.2.52431859.1517219938; SessionID=ID; __unam=7639673-16ef00748db-277e59b2-92; fpestid=ID; LtpaToken2=TOKEN; LtpaToken=TOKEN" "/local/notesdata/ADB/revenues.nsf"

```

Server time/timezone is under CEST, so the date was  
[root@server]# date  
Fri Apr 2 10:02:06 CEST 2021

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 2, 2021, 1:27pm UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/8 "2021-04-02T13:27:44Z")

</div>

So `02/Apr/2021:10:02:06 +0100` contains a timezone value and is stored in `apache.access.time`.

The thing is that `Fri Apr 2 10:02:06 CEST 2021` is actually `02/Apr/2021:10:02:06 +0200` as we switched from `CET` (GMT+1) to `CEST` (GMT+2) some days ago.

So I think that the problem is in the source. I mean in Apache logs.  
The Apache server is not generating the right information.

The log line should have been something like:

```auto
10.0.0.5 domain.com "CN=User/O=Company/C=CZ" [02/Apr/2021:10:02:06 +0200] "GET /ADB/revenues.nsf/icon11.gif HTTP/1.1" 200 1403 "https://domain.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" 139857020059654 "_ga=GA1.2.52431859.1517219938; SessionID=ID; __unam=7639673-16ef00748db-277e59b2-92; fpestid=ID; LtpaToken2=TOKEN; LtpaToken=TOKEN" "/local/notesdata/ADB/revenues.nsf"

```

---

<div class="post-metadata">

**Author:** ![radu990](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@radu990](https://discuss.elastic.co/u/radu990)\
**Post date:** [April 2, 2021, 1:39pm UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/9 "2021-04-02T13:39:42Z")

</div>

Thank you very much for your reply.  
I have just checked other servers with Apache which works ok in ELK/Kibana GUI. They have the following (please see the image below). So CEST is still with GMT +1 in logs, and they are displayed ok in ELK. I'm totally lost 🙂

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/4/d4faee09737319070b1cdf7571f9b05acbbac318.png)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 2, 2021, 1:51pm UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/10 "2021-04-02T13:51:08Z")

</div>

> [@radu990](#):
>
> I'm totally lost 🙂

I don't know...

But do we agree that `02/Apr/2021:10:02:06 +0100` is incorrect? If so, I'd try to fix that first.

---

<div class="post-metadata">

**Author:** ![radu990](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@radu990](https://discuss.elastic.co/u/radu990)\
**Post date:** [April 2, 2021, 1:59pm UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/11 "2021-04-02T13:59:37Z")

</div>

Yes, will try to change it to GMT +2 (+200) in the logs. Hopefully there will be changes afterwards.

---

<div class="post-metadata">

**Author:** ![radu990](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@radu990](https://discuss.elastic.co/u/radu990)\
**Post date:** [April 6, 2021, 8:24am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/12 "2021-04-06T08:24:14Z")

</div>

Thank you very much @dadoonet for the hint with the timezone. Changed to Africa/Cairo timezone instead of CEST, and now the logs are with +200 (GMT +2) and are working/displayed fine in ELK/Kibana.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 4, 2021, 8:24am UTC](https://discuss.elastic.co/t/1-hour-difference-on-showing-logs-in-elk-kibana/268744/13 "2021-05-04T08:24:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.