Logstash Date filter ISO8601 Issue

Elastic Stack Logstash
1

Hi there I am having some trouble with the date creation for my logs when they enter logstash, the end goal is to have the @timestamp field match the timestamp for the Auth or Syslog as thy are processed. These are in high precision format as exampled below. Please note the date match string is my throwing sepggiti at it as I have tired them on there own however nothing has run correctly so far, any help would be greatly received.

2018-03-07T17:05:11.274691+00:00 saltmaster sshd[3839]: Accepted publickey for vagrant from     10.0.2.2 port 46286 ssh2: RSA SHA256:GSlNfk3K8FA4m2R0cHsi10SxaH02qSOgaTZkH7dXnUc
2018-03-07T17:07:02.016188+00:00 saltmaster sshd[3870]: Received disconnect from 10.0.2.2 port 46286:11: disconnected by user
2018-03-07T17:07:02.016376+00:00 saltmaster sshd[3870]: Disconnected from 10.0.2.2 port 46286
2018-03-07T17:07:04.687096+00:00 saltmaster sshd[4541]: Accepted publickey for vagrant from 10.0.2.2 port 46320 ssh2: RSA SHA256:GSlNfk3K8FA4m2R0cHsi10SxaH02qSOgaTZkH7dXnUc
2018-03-07T17:17:09.548238+00:00 saltmaster sshd[4572]: Received disconnect from 10.0.2.2 port 46320:11: disconnected by user
2018-03-07T17:17:09.548420+00:00 saltmaster sshd[4572]: Disconnected from 10.0.2.2 port 46320
2018-03-07T17:17:12.293099+00:00 saltmaster sshd[5238]: Accepted publickey for vagrant from 10.0.2.2 port 46424 ssh2: RSA SHA256:GSlNfk3K8FA4m2R0cHsi10SxaH02qSOgaTZkH7dXnUc
2018-03-07T17:17:34.160069+00:00 saltmaster sshd[5269]: Received disconnect from 10.0.2.2 port 46424:11: disconnected by user
2018-03-07T17:17:34.160234+00:00 saltmaster sshd[5269]: Disconnected from 10.0.2.2 port 46424
2018-03-07T17:17:36.721722+00:00 saltmaster sshd[5305]: Accepted publickey for vagrant from 10.0.2.2 port 46426 ssh2: RSA SHA256:GSlNfk3K8FA4m2R0cHsi10SxaH02qSOgaTZkH7dXnUc

The following is a debug trace log from attempting to process logs, as you can see they are out about a second, I have also attempted to run old logs and they are not given the correct time stamp either.

[DEBUG] 2018-03-07 17:18:23.771 [nioEventLoopGroup-4-1] LoggingHandler - [id: 0xf0fb27b4, L:/192.168.60.11:5044 - R:/192.168.60.10:36820] FLUSH
{
        "system.auth.sudo.pwd" => "/home/vagrant",
                      "offset" => 3501,
                      "source" => "/var/log/client_logs/saltmaster/2018/03/07/auth/sudo.log",
                        "tags" => [
        [0] "auth",
        [1] "beats_input_codec_plain_applied"
    ],
       "system.auth.sudo.user" => "root",
                  "@timestamp" => 2018-03-07T17:18:22.221Z,
        "system.auth.hostname" => "saltmaster",
            "system.auth.user" => "vagrant",
    "system.auth.sudo.command" => "/bin/cat /var/log/client_logs/saltmaster/2018/03/07/syslog/sshd.log",
                    "@version" => "1",
                        "beat" => {
            "name" => "saltmaster",
        "hostname" => "saltmaster",
         "version" => "6.2.2"
    },
                        "host" => "saltmaster",
       "system.auth.timestamp" => "2018-03-07T17:18:19.054045+00:00",
        "system.auth.sudo.tty" => "pts/1"
}
{
               "@timestamp" => 2018-03-07T17:18:22.221Z,
     "system.auth.hostname" => "saltmaster",
      "system.auth.program" => "sudo",
                   "offset" => 3622,
                 "@version" => "1",
                     "beat" => {
            "name" => "saltmaster",
        "hostname" => "saltmaster",
         "version" => "6.2.2"
    },
                     "host" => "saltmaster",
      "system.auth.message" => "pam_unix(sudo:session): session opened for user root by vagrant(uid=0)",
                   "source" => "/var/log/client_logs/saltmaster/2018/03/07/auth/sudo.log",
    "system.auth.timestamp" => "2018-03-07T17:18:19.054901+00:00",
                     "tags" => [
        [0] "auth",
        [1] "beats_input_codec_plain_applied"
    ]
}
{
               "@timestamp" => 2018-03-07T17:18:22.221Z,
     "system.auth.hostname" => "saltmaster",
      "system.auth.program" => "sudo",
                   "offset" => 3725,
                 "@version" => "1",
                     "beat" => {
            "name" => "saltmaster",
        "hostname" => "saltmaster",
         "version" => "6.2.2"
    },
                     "host" => "saltmaster",
      "system.auth.message" => "pam_unix(sudo:session): session closed for user root",
                   "source" => "/var/log/client_logs/saltmaster/2018/03/07/auth/sudo.log",
    "system.auth.timestamp" => "2018-03-07T17:18:19.057285+00:00",
                     "tags" => [
        [0] "auth",
        [1] "beats_input_codec_plain_applied"
    ]
}

My test config should attached as I was informed my post is too large.

2 
date { match => [ "system.auth.timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ" ] }

should do it.

3

This input deals with all the timestamps are incoming, the item you suggest was very close just needed an extra Z on that one as the timestamp has the : in between the two

match => [ "system.auth.timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ", "yyyy-MM-dd'T'HH:mm:ssZZ", "yyyy-MM-dd'T'HH:mm:ss.SSSZZ", "ISO8601" ]

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.