Hi All,
Looking for some guidance, per the subject, I will give the solvee 50$ to show my appreciation. I will share solution on thread I don't care, but I will compensate.
Setup:
Rsyslog version 8.20
ES version 7.10
Rsyslog forwards all incoming logs to ES server, then handled by rsyslog to ES modules:
"mmjsonparse"
"omelasticsearch"
This is working well for us. We use the below template to dump into ES:
</>
"module(load="mmjsonparse") # for parsing CEE-enhanced syslog messages
module(load="omelasticsearch") # for outputting to Elasticsearch
#try to parse a structured log
action(type="mmjsonparse")
template(name="rsyslog-index" type="string" string="rsyslog-%$YEAR%.%$MONTH%.%$DAY%")
template(name="json-syslog" type="list") {
constant(value="{")
constant(value=""@timestamp":"") property(name="timereported" dateFormat="rfc3339")
constant(value="","host":"") property(name="hostname")
constant(value="","severity":"") property(name="syslogseverity-text")
constant(value="","facility":"") property(name="syslogfacility-text")
constant(value="","program":"") property(name="programname")
constant(value="","tag":"") property(name="syslogtag" format="json")
constant(value="",") property(name="$!all-json" position.from="2")
}
action(type="omelasticsearch" template="json-syslog" searchIndex="rsyslog-index" dynSearchIndex="on" bulkmode="on" errorfile="/var/log/omelastic_eror.log")
</>
"
Issue is the devices we are testing against will not send the entire date, ie no milliseconds, ends at HH:MM:SS .
We wanted to fix this by adding a timegenerated field to each record, using the unix timestamp of the system as the record. We added the mapping, "timegenerated" to the index. We have been trying to adjust the above template to correctly append the field, but everytime we try, it stops working ( guessing this means its not correct format/syntax ), remove field restart syslog and the index starts to accumulate the data immediately.
I thought for sure the below was the right syntax, but nothing seems to work:
( just appended to the top of the defined constants )
constant(value="\timegenerated":"%timegenerated:::date-rfc3339%") property=(name="timegenerated" dateFormat="rfc3339")
I have adjusted it many times but nothing has worked so far. Let me know if any more information is needed. I don't know the rules, if someone is gonna flame for offering 50$ for solution you can stuff it 