ESX logs -> remote syslog -> ELK : Timestamp problems

Elastic Stack Logstash
1

The syslog messages fromt the ESX server and being received on the remote syslog as below.

Jun 22 05:51:19 esxserv01 Vpxa: info vpxa[FFDF9B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-655fae54-a2] [VpxLRO] -- BEGIN task-internal-63125 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 526c483f-1107-81f8-85ca-d4b02c8ebea9
Jun 22 05:51:19 esxserv01 Vpxa: info vpxa[FFDF9B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-655fae54-a2] [VpxLRO] -- FINISH task-internal-63125

whereas the actual timestamp is as shown in the below log.

Jun 22 09:53:03 web_serv apache_access_log: 10.1.1.22- - [22/Jun/2017:09:53:03 +0400] "GET /" 200 72018 "-" "-"

The problem is when the ESX logs are indexed and I view them in Kibana the ESX logs are shown with timestamp at 05:51:19 instead of 09:51.

2

What's the timezone of the syslog messages, i.e. what's the timezone of "Jun 22 05:51:19"? What does your date filter look like? What's the system timezone of the machine where Logstash runs?

3

For example check out these logs. The apache log is parsed correctly whereas the one from ESX carries a 4 hour gap.

   {
        **"@timestamp" => 2017-06-22T08:58:48.000Z,**
              "host" => "10.1.1.22",
           "program" => "apache_access_log",
           "message" => "10.1.1.22 - - [22/Jun/2017:12:58:48 +0400] \"GET /\" 200 72018 \"-\" \"-\"",
         "logsource" => "web_serv",
         **"timestamp" => "Jun 22 12:58:48"**
    }
    {
          "prg_info" => "cpu6:33600",
          "err_type" => "error",
        **"@timestamp" => 2017-06-22T04:58:48.000Z,**
               "prg" => "ScsiDeviceIO",
           "err_msg" => "2651: Cmd(0x439e5b7e1a00) 0x1a, CmdSN 0x13b242 from world 0 to dev \"naa.60050760670323941fb4acae1be9e5c7\" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x0 0x0.",
              "host" => "10.2.2.33",
           "program" => "vmkernel",
           "message" => "cpu6:33600)ScsiDeviceIO: 2651: Cmd(0x439e5b7e1a00) 0x1a, CmdSN 0x13b242 from world 0 to dev \"naa.60050760670323941fb4acae1be9e5c7\" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x0 0x0.",
         "logsource" => "esxserv01",
         **"timestamp" => "Jun 22 08:58:48"**
    }
4 
[root@elk_server conf.d]# date
Thu Jun 22 14:01:18 GST 2017

we've hosted both the remote syslog and ELK in the same server. we're still testing :slight_smile:
I do not have a date filter. I did not need one as the messages are being received in syslog format.
They're formatted correctly from the Unix servers because they're being sent with the GST timezone.
What I suspect is the events from Unix are in GST and so they're corrected to UTC by reducing 4 hours.
But with ESX they're being sent in UTC and probably LS is reducing another 4 hours causing this.

5

So you're using a syslog input? You may have to replace it with a tcp input and grok and date filters to mimic what the syslog input does but passing timezone => "UTC" to the date filter so it doesn't assume that the timestamps is in local time.

1 Like
6

Thanks Magnus. I wanted to use only syslog input. I tried the below and it's working like a charm. Appreciate your help with this.

filter {
  if "esx" in [logsource]  {
date {
  match => [ "timestamp", "MMM dd HH:mm:ss" ]
  locale => "en"
  timezone => "UTC"
}
if ( [message] =~ "^verbose|^info|^-->|^crond" ) {
  drop { }
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.