ESX logs -> remote syslog -> ELK : Timestamp problems

The syslog messages fromt the ESX server and being received on the remote syslog as below.

Jun 22 05:51:19 esxserv01 Vpxa: info vpxa[FFDF9B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-655fae54-a2] [VpxLRO] -- BEGIN task-internal-63125 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 526c483f-1107-81f8-85ca-d4b02c8ebea9
Jun 22 05:51:19 esxserv01 Vpxa: info vpxa[FFDF9B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-655fae54-a2] [VpxLRO] -- FINISH task-internal-63125

whereas the actual timestamp is as shown in the below log.

Jun 22 09:53:03 web_serv apache_access_log: 10.1.1.22- - [22/Jun/2017:09:53:03 +0400] "GET /" 200 72018 "-" "-"

The problem is when the ESX logs are indexed and I view them in Kibana the ESX logs are shown with timestamp at 05:51:19 instead of 09:51.

What's the timezone of the syslog messages, i.e. what's the timezone of "Jun 22 05:51:19"? What does your date filter look like? What's the system timezone of the machine where Logstash runs?

For example check out these logs. The apache log is parsed correctly whereas the one from ESX carries a 4 hour gap.

   {
        **"@timestamp" => 2017-06-22T08:58:48.000Z,**
              "host" => "10.1.1.22",
           "program" => "apache_access_log",
           "message" => "10.1.1.22 - - [22/Jun/2017:12:58:48 +0400] \"GET /\" 200 72018 \"-\" \"-\"",
         "logsource" => "web_serv",
         **"timestamp" => "Jun 22 12:58:48"**
    }
    {
          "prg_info" => "cpu6:33600",
          "err_type" => "error",
        **"@timestamp" => 2017-06-22T04:58:48.000Z,**
               "prg" => "ScsiDeviceIO",
           "err_msg" => "2651: Cmd(0x439e5b7e1a00) 0x1a, CmdSN 0x13b242 from world 0 to dev \"naa.60050760670323941fb4acae1be9e5c7\" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x0 0x0.",
              "host" => "10.2.2.33",
           "program" => "vmkernel",
           "message" => "cpu6:33600)ScsiDeviceIO: 2651: Cmd(0x439e5b7e1a00) 0x1a, CmdSN 0x13b242 from world 0 to dev \"naa.60050760670323941fb4acae1be9e5c7\" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x0 0x0.",
         "logsource" => "esxserv01",
         **"timestamp" => "Jun 22 08:58:48"**
    }
[root@elk_server conf.d]# date
Thu Jun 22 14:01:18 GST 2017

we've hosted both the remote syslog and ELK in the same server. we're still testing :slight_smile:
I do not have a date filter. I did not need one as the messages are being received in syslog format.
They're formatted correctly from the Unix servers because they're being sent with the GST timezone.
What I suspect is the events from Unix are in GST and so they're corrected to UTC by reducing 4 hours.
But with ESX they're being sent in UTC and probably LS is reducing another 4 hours causing this.

So you're using a syslog input? You may have to replace it with a tcp input and grok and date filters to mimic what the syslog input does but passing timezone => "UTC" to the date filter so it doesn't assume that the timestamps is in local time.

1 Like

Thanks Magnus. I wanted to use only syslog input. I tried the below and it's working like a charm. Appreciate your help with this.

filter {
  if "esx" in [logsource]  {
date {
  match => [ "timestamp", "MMM dd HH:mm:ss" ]
  locale => "en"
  timezone => "UTC"
}
if ( [message] =~ "^verbose|^info|^-->|^crond" ) {
  drop { }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.