Log time doesnot match the @timestamp

[souravtyson]

Hi team,

I am trying to send the log from filebeat to elasticsearch. The log format is

0 0 37096 1198332 0 5635848 0 0 0 0 346 332 0 0 100 0 0 2018-09-28 15:52:100

POST _ingest/pipeline/_simulate
{ "pipeline" :{
"description": "Pipeline for parsing vmstat system logs.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"%{SPACE}%{NUMBER:vmstat.system.r}\s+%{NUMBER:vmstat.system.b}\s+%{NUMBER:vmstat.system.swpd}\s+%{NUMBER:vmstat.system.free}\s+%{NUMBER:vmstat.system.buff}\s+%{NUMBER:vmstat.system.cache}\s+%{NUMBER:vmstat.system.si}\s+%{NUMBER:vmstat.system.so}\s+%{NUMBER:vmstat.system.bi}\s+%{NUMBER:vmstat.system.bo}\s+%{NUMBER:vmstat.system.in}\s+%{NUMBER:vmstat.system.cs}\s+%{NUMBER:vmstat.system.us}\s+%{NUMBER:vmstat.system.sy}\s+%{NUMBER:vmstat.system.id}\s+%{NUMBER:vmstat.system.wa}\s+%{NUMBER:vmstat.system.st}\s+%{DATETIMESTAMP:vmstat.system.dateandtime}"
],
"pattern_definitions": {
"DATETIMESTAMP": "%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY} %{TIME}"
},
"ignore_missing": true
}
},{
"script" : {
"lang" : "painless",
"source" : "ctx.vmstat.system.cpuused=100-Integer.parseInt(ctx.vmstat.system.id);"
}
}
],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
},
"docs":[
{
"_source": {
"message": " 0 0 37096 1198332 0 5635848 0 0 0 0 346 332 0 0 100 0 0 2018-09-28 15:52:10"
}
}
]
}

my parser works and it gives me the output as accepted only my timestamp value is not matching.

{
"docs": [
{
"doc": {
"_index": "_index",
"_type": "_type",
"_id": "_id",
"_source": {
"vmstat": {
"system": {
"st": "0",
"b": "0",
"cache": "5635848",
"in": "346",
"sy": "0",
"bi": "0",
"wa": "0",
"cpuused": 0,
"bo": "0",
"swpd": "37096",
"cs": "332",
"r": "0",
"si": "0",
"id": "100",
"so": "0",
"free": "1198332",
"dateandtime": "2018-09-28 15:52:10",
"buff": "0",
"us": "0"
}
},
"message": " 0 0 37096 1198332 0 5635848 0 0 0 0 346 332 0 0 100 0 0 2018-09-28 15:52:10"
},
"_ingest": {
"timestamp": "2018-10-04T09:50:57.290Z"
}
}
}
]
}

is their any way to match timestamp and dateandtime variable. so that while selecting index in kibana,
i could search based on timestamp.

[jakelandis]

I think you will want to use the Date processor: https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html

Specifically add something like this after your grok processor (pick the appropriate timezone):

{
      "date" : {
        "field" : "vmstat.system.dateandtime",
        "formats" : ["yyyy-MM-dd HH:mm:ss"],
        "timezone": "Europe/Berlin"
      }
    }

[souravtyson]

tried your configuration as suggested.
here is what i am facing issue now. tried running my script in dev tools of kibana with following configuration. please find sceenshot attached.

getting following output

in the output you can see @timestamp variable matching with dateandtime variable. while their is one more timestamp variable with different value.

I also try to push the data to elasticsearch and my observation is completely different.

here you can see i am getting @timestamp field with completely different time value as compare to the value present in message field.

Note : my log server and elastic server are running in time sync.

so my question is why is timestamp field in _ingest field is different from other timing. It is showing time 5:30 minutes before and in kibana discover page i am seeing @timestamp value 5:30 minutes ahead of the message value

how can i make all these value same?

[jakelandis]

It is likely the lack of timezone information. When not provided, it defaults to the UTC (offset = 0). The system is likely logging using localized timestamps (e.g. not UTC timestamps) .

To fix this you will need to to configure the timezone in the date processor to match timezone of the log files, or change the system that is logging to log using UTC time.

[souravtyson]

This works for me.
I tried "timezone" : "Asia/Kolkata".