Logstash - how to parse rsyslog date field from logs message into @timestamp (UTC)

[Jenisha_Ramanathan]

Hi Team,

I am trying to monitor Linux logs with rsyslog and kibana. Able to receive the logs from rsyslog client to rsyslog server and load the data to elastic. Getting the reported time in kibana ( UTC format ) but we need to get the time from raw logs (EST format) and made it as global time filter.

Please find the below configuration used to load the data to elastic in the given json format and the logstash configuration.

json-template.conf:

template(name="json-template"
type="list") {
constant(value="{")
constant(value=""@timestamp":"") property(name="timereported" dateFormat="rfc3339")
constant(value="","@version":"1")
constant(value="","message":"") property(name="msg" format="json")
constant(value="","sysloghost":"") property(name="hostname")
constant(value="","severity":"") property(name="syslogseverity-text")
constant(value="","facility":"") property(name="syslogfacility-text")
constant(value="","programname":"") property(name="programname")
constant(value="","procid":"") property(name="procid")
constant(value=""}\n")
}

logstash.conf:

input {
udp {
host => "127.0.0.1"
port => 10514
codec => "json"
type => "rsyslog"
}
}
output {
if [type] == "rsyslog" {
elasticsearch {
hosts => ["http://10.67.169.18:9200"]
index => "rsyslog-logstash-%{+YYYY.MM.dd}"
user => "elastic"
password => "Infy123++"
}
}
}

Sample Output json for single document:

{
"_index": "rsyslog-logstash-2021.03.01",
"_type": "_doc",
"_id": "QiZe8XcBAmuLMuU_V1Et",
"_version": 1,
"_score": null,
"_source": {
"@version": "1",
"message": " I0302 00:17:07.513325 1667 kubelet_network_linux.go:111] Not using --random-fully in the MASQUERADE rule for iptables because the local version of iptables does not support it",
"facility": "daemon",
"host": "127.0.0.1",
"@timestamp": "2021-03-01T18:47:07.000Z",
"programname": "kubelet",
"sysloghost": "blb44imspro009",
"procid": "-",
"type": "rsyslog",
"severity": "info"
},
"fields": {
"@timestamp": [
"2021-03-01T18:47:07.000Z"
]
},

In the above sample example, i need to get the date from message in timestamp format.
Current output:
message - > I0302 00:17:07.513325
@timestamp -> 2021-03-01T18:47:07.000Z
Expected output :
message - > I0302 00:17:07.513325
@timestamp - > 2021-03-02T00:17:07.000Z

Could you kindly help me on how to extract the date filed from the input pattern and parse the field into @ timestamp.
Thank you in advance.

[Badger]

I have not tested it, but you could try

grok { match => { "message" => "(?<[@metadata][ts]>\d{4} \d{2}:\d{2}:\d{2})" } }
date { match => [ "[@metadata][ts]", "MMdd HH:mm:ss" ] }

See comments on guessing the year here.