Need help on the correct logstash.conf settings for processing a Log4Net file


(Harry Werkman) #1

Hi All.

I need to process a Log4Net file containing the following information;
############################################
LOG STARTED 2016-01-12 10:05:23,181
############################################
2016-01-28 00:00:00,000 [PhoenixInstance_Worker-8] DEBUG UNK0000T RetrieveAcceptorFilesJob - RetrieveAcceptorFiles: [Begin]
2016-01-28 00:00:00,000 [PhoenixInstance_Worker-8] INFO UNK0000O AcceptorFileRetrievalService - RWWA.Services.Proxies.AcceptorFileRetrieval.AutoRetrieveAcceptorFilesRequest : [Begin]
2016-01-28 00:00:00,000 [PhoenixInstance_Worker-8] INFO UNK0000O ConnectionWrapper - Request type RWWA.Services.Proxies.AcceptorFileRetrieval.AutoRetrieveAcceptorFilesRequest - Created an oracle connection
2016-01-28 00:00:00,000 [PhoenixInstance_Worker-8] DEBUG UNK0000T AutoRetrieveImpl - AutoRetrieveImpl: [Begin]
2016-01-28 00:00:00,000 [PhoenixInstance_Worker-8] INFO UNK0000O AutoRetrieveImpl - True : [Begin]
2016-01-28 00:00:00,000 [PhoenixInstance_Worker-8] INFO UNK0000O AutoRetrieveImpl - Request type System.Boolean - Created an oracle transaction
2016-01-28 00:00:00,016 [PhoenixInstance_Worker-8] DEBUG UNK0000T OracleAfrRepository - DbmsTryLock: dbms_lock.request: handle 1073741946107374194630
2016-01-28 00:00:00,016 [PhoenixInstance_Worker-8] DEBUG UNK0000T OracleAfrRepository - DbmsTryLock: dbms_lock.request: return 0
2016-01-28 00:00:00,016 [PhoenixInstance_Worker-8] INFO UNK0000O AutoRetrieveImpl - True : [End]
2016-01-28 00:00:00,016 [PhoenixInstance_Worker-8] INFO UNK0000O AutoRetrieveImpl - RWWA.Services.Proxies.AcceptorFileRetrieval.ListExternalDirectoryRequest : [Begin]
2016-01-28 00:00:00,016 [PhoenixInstance_Worker-8] INFO UNK0000O AutoRetrieveImpl - Request type RWWA.Services.Proxies.AcceptorFileRetrieval.ListExternalDirectoryRequest - Created an oracle transaction
2016-01-28 00:00:00,016 [PhoenixInstance_Worker-8] DEBUG UNK0000T FtpFileTransferAgent - [Ftp.ListDirectory] : ftp://rawdphc101v.rwwaq.com.au/AFR_FTP_TEST_AREA/
2016-01-28 00:00:00,109 [PhoenixInstance_Worker-8] DEBUG UNK0000T FtpFileTransferAgent - [Ftp.GetDateTimestamp] : ftp://rawdphc101v.rwwaq.com.au/AFR_FTP_TEST_AREA/20100322 Armidale NSW - Professional, FinalFields.xml
2016-01-28 00:00:00,109 [PhoenixInstance_Worker-8] DEBUG UNK0000T FtpFileTransferAgent - [Ftp.GetFileSize] : ftp://rawdphc101v.rwwaq.com.au/AFR_FTP_TEST_AREA/20100322 Armidale NSW - Professional, FinalFields.xml

As you can see there is one line per entry with a space separating each value on the first two values being a date and time.
I can ignore all items without a valid date and time value.

Any help would be greatly appreciated.

Regards,
Harry W.


(Magnus B├Ąck) #2

Have a look at http://grokconstructor.appspot.com/. It's a great help when writing grok expressions.


(system) #3