Nginx logs without year and seconds information

Elastic Stack Logstash
1

I have Nginx as loadbalancer which is generating logs without year and second information in timestamp. One of those logs are

08-10 09:28 root         ERROR    Error connecting to CRS REST API : [Errno 111] Connection refused
Error connecting to CRS REST API : [Errno 111] Connection refused

The pattern for this is :
(?m)%{MONTHNUM:monthNum}\-%{MONTHDAY:monthDay}\s*%{HOUR:hour}:%{MINUTE:minute}\s*%{WORD}\s*%{LOGLEVEL_CUSTOM:severity}\s*%{GREEDYDATA:messagePayload}

While I understand that year information can be done by Logstash with current year, which is fine with me as logs are not old and collected on daily basis, but as seconds part is important. I am not sure how to do that?
One additional thing, I am using date filter to convert it to time stamp to be stored in Elasticsearch which is as follows:

        mutate {
            add_field => { "timestamp" => "%{monthDay}%{monthNum} %{hour}:{minute}" }
        }

        date {
            match => ["timestamp","ddMM HH:mm", "YYYY-MM-dd HH:mm:ss,SSS", "YYYY-MM-dd HH:mm:ss", "YYYY/MM/dd HH:mm:ss", "dd/MMM/yyyy:HH:mm:ss Z", "ddMMYYYY HH:mm:ss", "ISO8601", "YYYY-MM-dd HH:mm:ss,SSSSSS", "YYYY-MM-dd HH:mm:ss.SSS", "YYYY-MM-dd HH:mm:ss.SSSSSS"]
            locale => "en"
            timezone => "UTC"
            target => "@timestamp"
        }

It is NOT parsing it correctly and time stamp is not same in Elasticsearch, not both date and time.
Any leads?

2

It is NOT parsing it correctly and time stamp is not same in Elasticsearch, not both date and time.

Please give a concrete example of such an event.

3

Yes, the time stamp in log and in Elasticsearch were different. Right now, I don't have that data as system was reset. Sorry for that. For example, in the log mentioned above the date was 10th Aug, and time was 09.28 UTC. but while querying the ES, the date was 11th Aug and time was different as well.

Date filter will see the field and try to match it with some formats mentioned there. So for me, ddMM HH:mm was not getting matched with "%{monthDay}%{monthNum} %{hour}:{minute}" .

I got the error just now, you can see in mutate filter
add_field => { "timestamp" => "%{monthDay}%{monthNum} %{hour}:{minute}"
% is missing from %{minute}. I have corrected it. Just going to test it.

One doubt is still there as what happens if the seconds part is missing from time.

4

One doubt is still there as what happens if the seconds part is missing from time.

Garbage in, garbage out. If the original data only has minute resolution I don't see how we could ever fix that in a reasonable way.

5

Yeah as seconds will not be as easy as year. Will see how system takes it and will add a reply here so that it can be used by any other user for base reference.

6

If the seconds information is not given in the log, logstash puts 00 in seconds place. Examples

"_source":{"device":"provo-topaz","devtype":"platform","host":"10.240.135.132","message":"08-18 06:18 requests.packages.urllib3.connectionpool INFO     Starting new HTTP connection (1): 10.240.136.88","offset":1455874,"part":"cc_horizon_10.240.135.132","parttype":"cc_horizon","path":"/opt/emc/caspian/logs/cc_horizon/config-sync.log","@version":"1","@timestamp":"2016-08-18T06:18:00.000Z","tags":[],"modulename":"requests.packages.urllib3.connectionpool","severity":"INFO","messagePayload":"Starting new HTTP connection (1): 10.240.136.88","orig_timestamp":"08-18 06:18","category":"AUDIT"},

 "_source":{"device":"orem-topaz","devtype":"platform","host":"10.240.135.134","message":"08-18 06:08 requests.packages.urllib3.connectionpool INFO     Starting new HTTP connection (1): 10.240.136.88","offset":1444922,"part":"cc_horizon_10.240.135.134","parttype":"cc_horizon","path":"/opt/emc/caspian/logs/cc_horizon/config-sync.log","@version":"1","@timestamp":"2016-08-18T06:08:00.000Z","tags":[],"modulename":"requests.packages.urllib3.connectionpool","severity":"INFO","messagePayload":"Starting new HTTP connection (1): 10.240.136.88","orig_timestamp":"08-18 06:08","category":"AUDIT"},

`"_source":{"device":"sandy-topaz","devtype":"platform","host":"10.240.135.133","message":"08-18 06:32 requests.packages.urllib3.connectionpool INFO     Starting new HTTP connection (1): 10.240.136.88","offset":1474487,"part":"cc_horizon_10.240.135.133","parttype":"cc_horizon","path":"/opt/emc/caspian/logs/cc_horizon/config-sync.log","@version":"1","@timestamp":"2016-08-18T06:32:00.000Z","tags":[],"modulename":"requests.packages.urllib3.connectionpool","severity":"INFO","messagePayload":"Starting new HTTP connection (1): 10.240.136.88","orig_timestamp":"08-18 06:32","category":"AUDIT"},`

I hope it helps.