Hi All,
I am pretty new to Logstash and I have an unstructured C++ log file from which I want to send only selected content of log to kafka topic from logstash. Below is the sample log file, can someone help with the config file that helps to achieve it.
I want the content which is within the xml tags (<DELPHI_REQUEST> ... </DELPHI_REQUEST> & <DELPHI_RESPONSE> .... </DELPHI_RESPONSE>)
12/20 00:02:07.47 ft_acce(28395)786 LogThread: TRDB_FT queue = 0 ( 0) [ 17, 0] XML
12/20 00:02:07.47 ft_acce(28395)786 LogThread: HNMNG_FT queue = 0 ( 0) [ 23, 0] XML
12/20 00:02:42.00 dte.cpp(28367)1914 IDL::ProcessRequestXML: DTE_KEY: Queueing request: SID:A232000005-, PERSISTID:47, WPID:0, SYS:NFAM, EMPID:, CMD:CLOSE_SESSION, PRIORITY:Default, TIMEOUT:300 sec
12/20 00:02:42.00 dte.cpp(28367)563 PendingResponseCount: Set(47) -> 1
12/20 00:02:42.00 priorit(28367)422 PutOnQueue: 47, "", defWait = 0, minWait = 0, timeoutInterval = 300
12/20 00:02:42.00 priorit(28367)457 PutOnQueue: Calling DequeueWakeUp() ...
12/20 00:02:42.00 priorit(28367)916 DequeueWakeUp: NULL
12/20 00:02:42.00 priorit(28363)703 Dequeue: return(47)
12/20 00:02:42.00 dte.cpp(28363)2386 ProcessRequest: DTE_KEY: Received XML request: SID:A232000005-, PERSISTID: 47, WPID:72000002, PRIORITY:Default, TIMEOUT:300 sec, returnIOR: IOR:010000001300000049444c3a4f7373526573756c74733a312e30000001000000000000007000000001010145100000003131332e3134302e3230372e31323900b78a203c1b00000014010f0052535459b6fb5d642d010002000000010000000300000056020000000000000008000000010800d0004f41540100000018000000010200d00100010001000000010001050901010000000000
<DELPHI_REQUEST>
<CTLHDR>
<DIPVER>5.0</DIPVER>
<DOMAIN>FTTP</DOMAIN>
<SYS_NAME>NFAM</SYS_NAME>
<DTIMEOUT>5</DTIMEOUT>
<ACKMNTS>N</ACKMNTS>
<MGRID>NFAM</MGRID>
</CTLHDR>
<REQUEST>
<SID>A232000005-</SID>
<REQUIRED>
<COMMAND>CLOSE_SESSION</COMMAND>
</REQUIRED>
</REQUEST>
</DELPHI_REQUEST>
12/20 06:40:51.37 dte.cpp(27102)3557 LogThread: Request queue = 0
12/20 06:40:51.37 dte.cpp(27102)3558 LogThread: Response queue = 0
12/20 06:40:51.37 ft_acce(27102)771 LogThread: FT response queue = 0
12/20 06:40:51.37 ft_acce(27102)773 LogThread: FT timeout queue = 0
12/20 06:42:10.90 dte.cpp(27087)1914 IDL::ProcessRequestXML: DTE_KEY: Queueing request: SID:, PERSISTID:1, WPID:76297808, SYS:VMOBILE, EMPID:, CMD:GET_HISTORY, PRIORITY:Demand, TIMEOUT:430 sec
12/20 06:42:10.90 dte.cpp(27087)563 PendingResponseCount: Set(1) -> 1
12/20 06:42:10.90 priorit(27087)422 PutOnQueue: 1, "", defWait = 0, minWait = 0, timeoutInterval = 430
12/20 06:42:10.90 priorit(27087)457 PutOnQueue: Calling DequeueWakeUp() ...
12/20 06:42:10.90 priorit(27087)916 DequeueWakeUp: NULL
12/20 06:42:10.90 priorit(27063)703 Dequeue: return(1)
12/20 06:42:10.90 dte.cpp(27063)2386 ProcessRequest: DTE_KEY: Received XML request: SID:, PERSISTID: 1, WPID:76297808, PRIORITY:Demand, TIMEOUT:430 sec, returnIOR: IOR:010000001300000049444c3a4f7373526573756c74733a312e30000001000000000000007000000001010150100000003131332e3134302e3230372e3132390071a900001b00000014010f00525354af42fc5d02c20a000200000001000000030000004402000000000000000800000001000000004f41540100000018000000010400680100010001000000010001050901010000000000
<DELPHI_REQUEST>
<CTLHDR>
<DIPVER>5.0</DIPVER>
<DOMAIN>FTTP</DOMAIN>
<SVC_ID>||null|EAST|GetFIOSTestHistoryFromDelphi|FTTP</SVC_ID>
<SVC_NAME>TEST</SVC_NAME>
<SYS_ID>WEBIFAS</SYS_ID>
<ATTACHMENTS>Y</ATTACHMENTS>
<SYS_NAME>VMOBILE</SYS_NAME>
<TSTMODE>CACHE</TSTMODE>
<USR_TYPE>NT</USR_TYPE>
<DTIMEOUT>PT420S</DTIMEOUT>
<REQ_TIME_STAMP>2019-12-20T06:42:10Z</REQ_TIME_STAMP>
<MGRID>NFAM</MGRID>
<TRANID>2019122076297808</TRANID>
<WPID>76297808</WPID>
<DTIMEOUTSEC>430</DTIMEOUTSEC>
<PRIORITY>Demand</PRIORITY>
</CTLHDR>
<REQUEST>
<RESEARCH_MAP>NNY</RESEARCH_MAP>
<USER_MAP>NNNNNYNN</USER_MAP>
<START_TIME/>
<END_TIME/>
<NETYPE>BHR</NETYPE>
<TN>4104857286</TN>
<TSTCOND>S</TSTCOND>
<REQUIRED>
<SVCTYPE>VOICE</SVCTYPE>
<COMMAND>GET_HISTORY</COMMAND>
<EMP_ID/>
<WK_TYPE>I</WK_TYPE>
<JOB_TYPE>M</JOB_TYPE>
</REQUIRED>
</REQUEST>
</DELPHI_REQUEST>
12/20 06:42:10.91 dte.cpp(27063)2449 ProcessRequest: DTE_KEY: Created new session: SID:A697000000-, SYS:VMOBILE, EMPID:
12/20 06:42:10.91 dte.cpp(27063)2523 ProcessRequest: persistID: 1, assigned to new session A697000000-
12/20 06:42:10.92 fios.cp(27063)9027 FIOSSession::DecodeAndValidate() SID:A697000000- Clearing the session
12/20 07:46:23.63 priorit(27066)916 DequeueWakeUp: NULL
12/20 07:46:23.63 priorit(27041)703 Dequeue: return(697000010)
12/20 07:46:23.63 ft_acce(27041)1836 FTResponseThread: DTE_KEY: SID:A697000004-, WPID:76297812, COMMAND:MANAGE-LR-CACHE-COMMAND, FT:CACHE_FT, REQID:697000010
12/20 07:46:23.63 ft_acce(27041)1845 DTE_KEY: SID:A697000004-, WPID:76297812, COMMAND:MANAGE-LR-CACHE-COMMAND, REQID:697000010,
<DELPHI_RESPONSE>
<CTLHDR>
<SVC_NAME>TEST</SVC_NAME>
<DOMAIN>FTTP</DOMAIN>
<SYS_NAME>DTI_EXPRESS</SYS_NAME>
<SVC_ID>|3211039005005911000|HEM</SVC_ID>
<SYS_ID>10-118-224-197.ebiz.verizon.com</SYS_ID>
<DTIMEOUTSEC>2</DTIMEOUTSEC>
<STATMSG>Y</STATMSG>
<MULTRES>Y</MULTRES>
<ATTACHMENTS>Y</ATTACHMENTS>
<REQ_TIME_STAMP>2019-12-20T07:46:20Z</REQ_TIME_STAMP>
<PRIORITY>Demand</PRIORITY>
<MGRID>697000010</MGRID>
<WPID>76297812</WPID>
</CTLHDR>
<REQUEST>
<COMMAND>RETRIEVE_CACHE_INFO</COMMAND>
<EMP_ID>SIVARA2</EMP_ID>
<DURATION>86400</DURATION>
<WK_TYPE>M</WK_TYPE>
<ATTRIBUTES>
<NUM_ATTRS>2</NUM_ATTRS>
<ATTRIBUTE>
<ATTR_NAME>CACHE_TYPE</ATTR_NAME>
<ATTR_VAL>FTTP_PATH_INFO</ATTR_VAL>
</ATTRIBUTE>
<ATTRIBUTE>
<ATTR_NAME>KEY</ATTR_NAME>
<ATTR_VAL>7815196969</ATTR_VAL>
</ATTRIBUTE>
</ATTRIBUTES>
</REQUEST>
<RESULTS>
<RETC>1010</RETC>
<RSTYPE>E</RSTYPE>
<INFOMSG>Cache data not found</INFOMSG>
<ERRMSG>Cache data not found</ERRMSG>
<RESP_TIME_STAMP>2019-12-20T07:46:23Z</RESP_TIME_STAMP>
</RESULTS>
</DELPHI_RESPONSE>
12/20 07:46:23.63 ft_acce(27041)1854 FTResponseThread: Generating dequeue event: CACHE_FT
12/20 07:46:23.63 priorit(27041)916 DequeueWakeUp: CACHE_FT
12/20 07:46:23.63 dte.cpp(27041)3104 ProcessCommandResults: DTE_KEY: SID:A697000004-, FIOS-PATHINFO-CACHE-OUTAGE-STATE::MANAGE-LR-CACHE-COMMAND completed (1/1)
12/20 07:46:23.63 dte.cpp(27041)3144 ProcessCommandResults: DTE_KEY: SID:A697000004-, transition FIOS-PATHINFO-CACHE-OUTAGE-STATE -> FIOS-TEST-HISTORY-ARCHIVE-STATE
12/20 07:46:23.63 fios_re(27041)4537 FIOSRemarks::PrepareFinalResponse() WPID:76297812 using FIOS-RESULT-FCS=ALL
12/20 07:46:23.63 fios.cp(27041)6655 GetNeedSummaryByCommand WPID: 76297812, VOICE+GET_TOPOLOGY+SUMMARY = 0
12/20 07:46:23.63 fios.cp(27041)28559 PreparePCAN() WPID:76297812 prepared PCAN=7815196969
12/20 07:46:23.63 dte.cpp(27041)3276 RunState: DTE_KEY: SID:A697000004-, FIOS-TEST-HISTORY-ARCHIVE-STATE => ARCHIVE-TEST-HISTORY-COMMAND
12/20 07:46:23.63 ft_acce(27041)991 QueueCommand: DTE_KEY: SID:A697000004-, WPID:76297812, COMMAND:ARCHIVE-TEST-HISTORY-COMMAND, FT:TRDB_FT (XML), OSSID:TRDB, REQID:697000011, PRIORITY:Demand, TIMEOUT:15 sec