Problem with the time in elastic

Elastic Stack Elasticsearch
1

hi i have elasticsearch-7.10.1 and kibana-7.10.1 and filebeat-7.10.1
and i cant configure the time for israel
in the Discover i see the timestamp and log_date are different.
how i can fix that the both time are Will be the same time.
thanks

Screenshot 2022-08-24 151530
Screenshot 2022-08-24 1515301577×205 40.8 KB

2

some one?

3

Timestamps in Elasticsearch are always stored in UTC timezone. Kibana does by default translate the main timestamp to the local timezone, which I suspect is what you see in effect here. Please retrieve the raw JSON document and show it here.

4

{
"_index": "x-2022.08.28",
"_type": "_doc",
"_id": "lDPX5YIBQlgtrhReqQ55",
"_version": 1,
"_score": null,
"_source": {
"server_name": "x",
"agent": {
"hostname": "ELKSVR",
"name": "ELKSVR",
"id": "cf92b3a0-145e-4cf5-8d06-570bce700b79",
"type": "filebeat",
"ephemeral_id": "52236d9a-729d-456e-9cd8-c57308bec428",
"version": "7.10.1"
},
"deal": "87",
"run_id": "3",
"log": {
"file": {
"path": "x"
},
"offset": 10587478
},
"system_name": "x",
"log_level": "Info",
"message": "Finished:OK, with Warnings",
"input": {
"type": "log"
},
"@timestamp": "2022-08-28T22:04:46.000+03:00",
"ecs": {
"version": "1.6.0"
},
"log_date": "08/28/2022 22:04:46",
"stage_name": "x",
"host": {
"hostname": "ELKSVR",
"os": {
"build": "14393.4886",
"kernel": "x (rs1_release.220104-1735)",
"name": "Windows Server 2016 Standard",
"family": "windows",
"version": "10.0",
"platform": "windows"
},
"ip": [
"fe80::65ef:5422:fc24:a89f",
"192.168.0.198",
"fe80::5efe:c0a8:c6"
],
"name": "ELKSVR",
"id": "6fd8c9ec-e901-422d-87fe-18fba7ca496a",
"mac": [
"00:15:5d:01:85:05",
"00:00:00:00:00:00:00:e0"
],
"architecture": "x86_64"
},
"fields": {
"pipeline": "xxxxx",
"index": "xxxxx"
}
},
"fields": {
"@timestamp": [
"2022-08-28T19:04:46.000Z"
],
"log_date": [
"2022-08-28T22:04:46.000Z"
]
},
"sort": [
1661713486000
]
}

5

some one?

6

Those are indeed different times, what does they look in the document?

The difference in those times are +0300, but without knowing how they look in the source document before filebeat is not possible to know which one is wrong and why.

How does the source document looks like? How are you indexing your data, are you using any filebeat module or it is a custom log with a custom ingest pipeline?

7

custom ingest pipeline
i use Processors wite grok

[
{
"grok": {
"field": "message",
"patterns": [
"%{DATESTAMP:log_date}\t%{DATA:log_level}\t%{DATA:stage_name}\t%{DATA:server_name}\t%{DATA:system_name}\tDealCode %{NUMBER:deal}\tPPBRunID %{NUMBER:run_id}\tDG %{NUMBER:dg}\tRunName %{DATA:runner_name}\t\[\[\[%{DATA:message}\]\]\]",
"%{DATESTAMP:log_date}\t%{DATA:log_level}\t%{DATA:stage_name}\t%{DATA:server_name}\t%{DATA:system_name}\tDealCode %{NUMBER:deal}\tPPBRunID %{NUMBER:run_id}\tDG %{NUMBER:dg}\t\t\[\[\[%{DATA:message}\]\]\]",
"%{DATESTAMP:log_date}\t%{DATA:log_level}\t%{DATA:stage_name}\t%{DATA:server_name}\t%{DATA:system_name}\tDealCode %{NUMBER:deal}\tPPBRunID %{NUMBER:run_id}\t\t\t\[\[\[%{DATA:message}\]\]\]",
"%{DATESTAMP:log_date}\t%{DATA:log_level}\t%{DATA:stage_name}\t%{DATA:server_name}\t%{DATA:system_name}\tDealCode %{NUMBER:deal}\tPPBRunID %{NUMBER:run_id}\t\tRunName %{DATA:runner_name}\t\[\[\[%{DATA:message}\]\]\]"
]
}
},
{
"date": {
"field": "log_date",
"formats": [
"MM/dd/yyyy HH:mm:ss"
]
}
}
]

8

You also need to share the raw message that filebeat is consuming, it is not possible to know what is the wrong date without seeing the original message.

But from your date filter your date string does not have any information about timezone, if your original date is in your local timezone, which is UTC + 0300, you need to specify this in your date processor, if you do not specify a timezone it will assume that the date string is already in UTC and this can lead to wrong times in Kibana.

It should be something like this:

    {
      "date" : {
        "field" : "log_date",
        "formats" : ["MM/dd/yyyy HH:mm:ss"],
        "timezone" : "Asia/Jerusalem
      }
    }
9

i dont understand , its configure like this
{
"date" : {
"field" : "log_date",
"formats" : ["MM/dd/yyyy HH:mm:ss"],
"timezone" : "Asia/Jerusalem
}
}

10

How are filebeat receiving the logs? It is reading from a file?

You need to shared the source message before you run the ingest pipeline, before filebeat reading it.

As I said, your date string does not have any information about the timezone, if your date string is in your local time, which is Asia/Jerusalem, you need to tell the date processer in the ingest pipeline to use this timezone, if you do not configure a timezone, this date will be interpreted as already being in UTC.

Did you change the date processor in your ingest pipeline added the timezone? Did the time difference still present?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.