Grok pattern to add some metadata to every log

[Minor_Vargas]

Hello all,

I wrote a grok pattern that matches the beginning of each log line (Writing table data). I need to take the timestamp that I already matched and append it at the beginning of each log line. I'm facing 2 issues; The first one is that it's only matching the first result, and the second one is that I don't know how to append it.. This is to create a custom ingest pipeline. The grok pattern is this:

"(?m)%{DATESTAMP_OTHER:timestamp}%{GREEDYDATA} (?<PWD>pwd=)%{UNIXPATH:path}%{GREEDYDATA}(?<LOGSTART>Writing table data)%{GREEDYDATA}"

and the logs file goes more or like this

Sun Dec 13 08:00:15 CST 2020: Starting docker, stream=prod, version=latest, pwd=/home/cldap/tools/prod/CapacityWriter
adding java runtime options: -Xmx4096m -Xms2048m
Writing table data, snapshotId=364596, siteName=Singapore, podId=2, datacenter=datacenter, esxName=esxName
Exception thrown while executing class SQLURL.com: DB2 SQL Error: SQLCODE=-551, SQLSTATE=42501, SQLERRMC=CLDCAPAPPL;SELECT;CLDCAP.CLDCAPX_VMINFO_LATEST, DRIVER=4.23.42
    Failed to write data to kafka, streamName=xseries.vminfo, java.lang.Exception: SQL failed to read data, SELECT * FROM CLDCAPX_VMINFO_LATEST WHERE SNAPSHOT_ID=?
Writing table data, snapshotId=364596, siteName=Singapore, podId=2, datacenter=datacenter, esxName=esxName
Exception thrown while executing class SQLURL.com: DB2 SQL Error: SQLCODE=-551, SQLSTATE=42501, SQLERRMC=CLDCAPAPPL;SELECT;CLDCAP.CLDCAPX_VMINFO_LATEST, DRIVER=4.23.42
    Failed to write data to kafka, streamName=xseries.vminfo, java.lang.Exception: SQL failed to read data, SELECT * FROM CLDCAPX_VMINFO_LATEST WHERE SNAPSHOT_ID=?
Sun Dec 13 08:01:45 CST 2020:  Finished, rc=0

[Christian_Dahlqvist]

Grok is used to find and extract data - it does not modify it. You probably therefore need to extract the data using the grok filter and combine it with some other filter(s) to append.