How to extract only required content from a logfile and send it to filter parsing in logstash config file

Hi All,

I am pretty new to Logstash and I have an unstructured C++ log file from which I want to send only selected content of log to kafka topic from logstash. Below is the sample log file, can someone help with the config file that helps to achieve it.

I want the content which is within the xml tags (<DELPHI_REQUEST> ... </DELPHI_REQUEST> & <DELPHI_RESPONSE> .... </DELPHI_RESPONSE>)

12/20 00:02:07.47 ft_acce(28395)786  LogThread:        TRDB_FT queue =    0 (   0) [    17,      0] XML
      12/20 00:02:07.47 ft_acce(28395)786  LogThread:       HNMNG_FT queue =    0 (   0) [    23,      0] XML
      12/20 00:02:42.00 dte.cpp(28367)1914 IDL::ProcessRequestXML: DTE_KEY: Queueing request: SID:A232000005-, PERSISTID:47, WPID:0, SYS:NFAM, EMPID:, CMD:CLOSE_SESSION, PRIORITY:Default, TIMEOUT:300 sec
      12/20 00:02:42.00 dte.cpp(28367)563  PendingResponseCount: Set(47) -> 1
      12/20 00:02:42.00 priorit(28367)422  PutOnQueue: 47, "", defWait = 0, minWait = 0, timeoutInterval = 300
      12/20 00:02:42.00 priorit(28367)457  PutOnQueue: Calling DequeueWakeUp() ...
      12/20 00:02:42.00 priorit(28367)916  DequeueWakeUp: NULL
      12/20 00:02:42.00 priorit(28363)703  Dequeue: return(47)
      12/20 00:02:42.00 dte.cpp(28363)2386 ProcessRequest: DTE_KEY: Received XML request: SID:A232000005-, PERSISTID: 47, WPID:72000002, PRIORITY:Default, TIMEOUT:300 sec, returnIOR: IOR:010000001300000049444c3a4f7373526573756c74733a312e30000001000000000000007000000001010145100000003131332e3134302e3230372e31323900b78a203c1b00000014010f0052535459b6fb5d642d010002000000010000000300000056020000000000000008000000010800d0004f41540100000018000000010200d00100010001000000010001050901010000000000
<DELPHI_REQUEST>
 <CTLHDR>
  <DIPVER>5.0</DIPVER>
  <DOMAIN>FTTP</DOMAIN>
  <SYS_NAME>NFAM</SYS_NAME>
  <DTIMEOUT>5</DTIMEOUT>
  <ACKMNTS>N</ACKMNTS>
  <MGRID>NFAM</MGRID>
 </CTLHDR>
 <REQUEST>
  <SID>A232000005-</SID>
  <REQUIRED>
   <COMMAND>CLOSE_SESSION</COMMAND>
  </REQUIRED>
 </REQUEST>
</DELPHI_REQUEST>
12/20 06:40:51.37 dte.cpp(27102)3557 LogThread:        Request queue = 0
      12/20 06:40:51.37 dte.cpp(27102)3558 LogThread:       Response queue = 0
      12/20 06:40:51.37 ft_acce(27102)771  LogThread:    FT response queue = 0
      12/20 06:40:51.37 ft_acce(27102)773  LogThread:     FT timeout queue = 0
      12/20 06:42:10.90 dte.cpp(27087)1914 IDL::ProcessRequestXML: DTE_KEY: Queueing request: SID:, PERSISTID:1, WPID:76297808, SYS:VMOBILE, EMPID:, CMD:GET_HISTORY, PRIORITY:Demand, TIMEOUT:430 sec
      12/20 06:42:10.90 dte.cpp(27087)563  PendingResponseCount: Set(1) -> 1
      12/20 06:42:10.90 priorit(27087)422  PutOnQueue: 1, "", defWait = 0, minWait = 0, timeoutInterval = 430
      12/20 06:42:10.90 priorit(27087)457  PutOnQueue: Calling DequeueWakeUp() ...
      12/20 06:42:10.90 priorit(27087)916  DequeueWakeUp: NULL
      12/20 06:42:10.90 priorit(27063)703  Dequeue: return(1)
      12/20 06:42:10.90 dte.cpp(27063)2386 ProcessRequest: DTE_KEY: Received XML request: SID:, PERSISTID: 1, WPID:76297808, PRIORITY:Demand, TIMEOUT:430 sec, returnIOR: IOR:010000001300000049444c3a4f7373526573756c74733a312e30000001000000000000007000000001010150100000003131332e3134302e3230372e3132390071a900001b00000014010f00525354af42fc5d02c20a000200000001000000030000004402000000000000000800000001000000004f41540100000018000000010400680100010001000000010001050901010000000000
<DELPHI_REQUEST>
 <CTLHDR>
  <DIPVER>5.0</DIPVER>
  <DOMAIN>FTTP</DOMAIN>
  <SVC_ID>||null|EAST|GetFIOSTestHistoryFromDelphi|FTTP</SVC_ID>
  <SVC_NAME>TEST</SVC_NAME>
  <SYS_ID>WEBIFAS</SYS_ID>
  <ATTACHMENTS>Y</ATTACHMENTS>
  <SYS_NAME>VMOBILE</SYS_NAME>
  <TSTMODE>CACHE</TSTMODE>
  <USR_TYPE>NT</USR_TYPE>
  <DTIMEOUT>PT420S</DTIMEOUT>
  <REQ_TIME_STAMP>2019-12-20T06:42:10Z</REQ_TIME_STAMP>
  <MGRID>NFAM</MGRID>
  <TRANID>2019122076297808</TRANID>
  <WPID>76297808</WPID>
  <DTIMEOUTSEC>430</DTIMEOUTSEC>
  <PRIORITY>Demand</PRIORITY>
 </CTLHDR>
 <REQUEST>
  <RESEARCH_MAP>NNY</RESEARCH_MAP>
  <USER_MAP>NNNNNYNN</USER_MAP>
  <START_TIME/>
  <END_TIME/>
  <NETYPE>BHR</NETYPE>
  <TN>4104857286</TN>
  <TSTCOND>S</TSTCOND>
  <REQUIRED>
   <SVCTYPE>VOICE</SVCTYPE>
   <COMMAND>GET_HISTORY</COMMAND>
   <EMP_ID/>
   <WK_TYPE>I</WK_TYPE>
   <JOB_TYPE>M</JOB_TYPE>
  </REQUIRED>
 </REQUEST>
</DELPHI_REQUEST>
      12/20 06:42:10.91 dte.cpp(27063)2449 ProcessRequest: DTE_KEY: Created new session: SID:A697000000-, SYS:VMOBILE, EMPID:
      12/20 06:42:10.91 dte.cpp(27063)2523 ProcessRequest: persistID: 1, assigned to new session A697000000-
      12/20 06:42:10.92 fios.cp(27063)9027 FIOSSession::DecodeAndValidate() SID:A697000000- Clearing the session
	  12/20 07:46:23.63 priorit(27066)916  DequeueWakeUp: NULL
      12/20 07:46:23.63 priorit(27041)703  Dequeue: return(697000010)
      12/20 07:46:23.63 ft_acce(27041)1836 FTResponseThread: DTE_KEY: SID:A697000004-, WPID:76297812, COMMAND:MANAGE-LR-CACHE-COMMAND, FT:CACHE_FT, REQID:697000010
      12/20 07:46:23.63 ft_acce(27041)1845 DTE_KEY: SID:A697000004-, WPID:76297812, COMMAND:MANAGE-LR-CACHE-COMMAND, REQID:697000010, 
	  <DELPHI_RESPONSE>
 <CTLHDR>
  <SVC_NAME>TEST</SVC_NAME>
  <DOMAIN>FTTP</DOMAIN>
  <SYS_NAME>DTI_EXPRESS</SYS_NAME>
  <SVC_ID>|3211039005005911000|HEM</SVC_ID>
  <SYS_ID>10-118-224-197.ebiz.verizon.com</SYS_ID>
  <DTIMEOUTSEC>2</DTIMEOUTSEC>
  <STATMSG>Y</STATMSG>
  <MULTRES>Y</MULTRES>
  <ATTACHMENTS>Y</ATTACHMENTS>
  <REQ_TIME_STAMP>2019-12-20T07:46:20Z</REQ_TIME_STAMP>
  <PRIORITY>Demand</PRIORITY>
  <MGRID>697000010</MGRID>
  <WPID>76297812</WPID>
 </CTLHDR>
 <REQUEST>
  <COMMAND>RETRIEVE_CACHE_INFO</COMMAND>
  <EMP_ID>SIVARA2</EMP_ID>
  <DURATION>86400</DURATION>
  <WK_TYPE>M</WK_TYPE>
  <ATTRIBUTES>
   <NUM_ATTRS>2</NUM_ATTRS>
   <ATTRIBUTE>
    <ATTR_NAME>CACHE_TYPE</ATTR_NAME>
    <ATTR_VAL>FTTP_PATH_INFO</ATTR_VAL>
   </ATTRIBUTE>
   <ATTRIBUTE>
    <ATTR_NAME>KEY</ATTR_NAME>
    <ATTR_VAL>7815196969</ATTR_VAL>
   </ATTRIBUTE>
  </ATTRIBUTES>
 </REQUEST>
 <RESULTS>
  <RETC>1010</RETC>
  <RSTYPE>E</RSTYPE>
  <INFOMSG>Cache data not found</INFOMSG>
  <ERRMSG>Cache data not found</ERRMSG>
  <RESP_TIME_STAMP>2019-12-20T07:46:23Z</RESP_TIME_STAMP>
 </RESULTS>
</DELPHI_RESPONSE>
      12/20 07:46:23.63 ft_acce(27041)1854 FTResponseThread: Generating dequeue event: CACHE_FT
      12/20 07:46:23.63 priorit(27041)916  DequeueWakeUp: CACHE_FT
      12/20 07:46:23.63 dte.cpp(27041)3104 ProcessCommandResults: DTE_KEY: SID:A697000004-, FIOS-PATHINFO-CACHE-OUTAGE-STATE::MANAGE-LR-CACHE-COMMAND completed (1/1)
      12/20 07:46:23.63 dte.cpp(27041)3144 ProcessCommandResults: DTE_KEY: SID:A697000004-, transition FIOS-PATHINFO-CACHE-OUTAGE-STATE -> FIOS-TEST-HISTORY-ARCHIVE-STATE
      12/20 07:46:23.63 fios_re(27041)4537 FIOSRemarks::PrepareFinalResponse() WPID:76297812 using FIOS-RESULT-FCS=ALL
      12/20 07:46:23.63 fios.cp(27041)6655 GetNeedSummaryByCommand WPID: 76297812, VOICE+GET_TOPOLOGY+SUMMARY = 0
      12/20 07:46:23.63 fios.cp(27041)28559 PreparePCAN() WPID:76297812 prepared PCAN=7815196969
      12/20 07:46:23.63 dte.cpp(27041)3276 RunState: DTE_KEY: SID:A697000004-, FIOS-TEST-HISTORY-ARCHIVE-STATE => ARCHIVE-TEST-HISTORY-COMMAND 
      12/20 07:46:23.63 ft_acce(27041)991  QueueCommand: DTE_KEY: SID:A697000004-, WPID:76297812, COMMAND:ARCHIVE-TEST-HISTORY-COMMAND, FT:TRDB_FT (XML), OSSID:TRDB, REQID:697000011, PRIORITY:Demand, TIMEOUT:15 sec

Can anyone help me out here?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.