How to Json log format output for my system log

Hi,

I have my log in below format, it actually has all fields delimited by ^. However, in between we can have few other lines which are nothing but the field headers. I am unable to get this log converted into json format using logstash like below:

Expected Json Format:
{"SESSIONID":"A335000000-","TRANID":"2020091676094508","WPID":"76094508","SYSNAME ":"VRP" and so on ....}

Can anyone help me hot to achieve this as it is very much needed for me.

System Log Format:

-=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=-
@ 09/16 06:38:16.46 @@start@@
$
      09/16 06:38:16.46 rw/dte/(24538)3218
01 SESSIONID    02 TRANID      03 WPID         04 SYSNAME    05 CID         06 TN
07 ORDERNO      08 COMMAND     09 SVCTYPE      10 WKTYPE     11 HOST        12 IN
13 OUT          14 DIFF        15 NOOF ANA     16 ANALYSIS   17 NOOF CMD    18 COMMANDS
19 EMPID        20 NID         21 ATTRIBUTES   22 CACHE      23 CKTMODEL    24 TSYS
25 TSTMODE      26 CACH ARCH R 27 CACH RTRV R  28 RETC
16 ANALYSIS EXPANSION
01 DC           02 EXP         03 TRBLCODE     04 ACTION     05 STB SER     06 NETYPE
07 NEID
17 COMMAND  EXPANSION
01 ID           02 INSTANCE    03 STATENAME    04 CMDFMLY    05 START       06 END
17 COMMAND  RESULT EXPANSION
01 NAME         02 INSTANCE    03 INTVAL       04 ERRCODE    05 ERRMSG      06 USERFVAL
07 TRBLCODE     08 NESTR       09 COMMENT      10 EXP


      09/16 06:38:16.47 rw/dte/(24538)3303 A335000000-^2020091676094508^76094508^VRP^22/VXXA/12345    /VZMA^^^QT::ASO^VIDEO^M^dhost1^06:36:58^06:38:16^78^1^:ANASTART::ANAS:RVW~Systemunable to communicate with Tellabs Element Management System. Access Guardian Error: Unable to open connection. Please retry this request later.~TSAFCACF~Retry, follow M&P~~ONT~[PENDING INSTALL]~~:ANE::ANAEND:^15^:CMDSTART::CMDS:ID=MANAGE-LR-CACHE-COMMAND~INS=RETRIEVE~ST=FIOS-PATHINFO-CACHE-STATE~CF=NETWORK~ST=1600238218~ED=1600238218~FT=CACHE_FT~OS=CACHESRV:RESSTART::RESEND::CMDE::CMDS:ID=GET-CSD2~INS=1~ST=FIOS-PATHINFO-INIT-STATE~CF=NETWORK~ST=1600238218~ED=1600238256~FT=IVIEW_FT~OS=IVIEW_FT:RESSTART::RESEND::CMDE::CMDS:ID=GET~INS=7812516069~ST=FIOS-PATHINFO-STATEA~CF=NETWORK~ST=1600238256~ED=1600238279~RESSTART::RESEND::CMDE::CMDS:ID=GET-CSD2~INS=7813205883~ST=FIOS-PATHINFO-STATEA~CF=NETWORK~ST=1600238256~ED=1600238279~FT=IVIEW_FT~OS=RESEND::CMDE::CMDS:ID=GET-er::CMDE::CMDS:ID=FIOS-FINAL-COMMAND-PATHINFO~INS=~ST=FIOS-PATHINFO-END-STATE~CF=NETWORK~ST=1600238294~ED=1600238294~FT=INVALID_FT~OS=:RESSTART::RESEND::CMDE::CMDS:ID=RETRIEVE-TEST-HISTORY-COMMAND~INS=CACHE_RESULTS~ST=FIOS-RETRIEVE-CACHE-RESULTS-STATE-A~CF=RT::RESS:N=CACHE-HISTORY-OK-RESULT~I=CACHE_RESULTS~V=NO-RESULTS~EC=~ED=~U=Retrieve Cache History Command Successful~C=TSDELTOK~CKT=~CMT=1~EXP=0:RESE::RESEND::CMDE::CMDS:ID=RTRV-FT-RESP-COMMAND~INS=RTRV-FT-RESP~ST=FIOS-RETRIEVE-CACHE-RESULTS-STATE-B~CF=NETWORK~ST=~ED=~FT=IGNITE_FT~OS=::RESEND::CMDE::CMDS:ID=RETRV-IGNITE-GENERIC-KEY-EXECUTION-STATUS-COMMAND~INS=RETRV~ST=FIOS-RETRIEVE-INPROGRESS-IG-STATE-C~CF=NETWORK~ST=1600238294~ED=1600238294~FT=IGNITE_FT~OS=IGNITE:RESSTART::RESEND::CMDE::CMDS:ID=RETRIEVE-PON-ALARM~INS=~ST=FIOS-VODRF-INIT-VER1-STATE~CF=NETWORK~ST=1600238294~ED=1600238294~FT=PONNMS_FT~OS=WSG_PONNMS:RESSTART::RESS:N=OLT-ALARMS-INCOMPLETE-RESULT~I=~V=~EC=FAILED~ED=FAILED~U=FAILED~C=TSDELRVW~CKT=ONT:Pending_7_ONT,X_1V_DATA,1.1.1,LET-8-5-1-1V,AFCM,ONT612,,SFU,,YES,[PENDING INSTALL],,,25 Mbps Mpbs,25 Mbps,25 Mbps,25 Mbps Mpbs,25 Mbps,25 Mbps:~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDS:ID=TELLABS-VIDEO-PORT-STATUS~INS=~ST=FIOS-VODRF-INIT-VER1-STATE~CF=NETWORK~ST=1600238294~ED=1600238294~FT=PONEMS_FT~OS=PONEMS:RESSTART::RESS:N=VIDEO-PORT-STATUS-INCOMPLETE-RESULT~I=~V=~EC=FT-EXCEPTION-SEND-FAILED~ED=PrimarySecondarybothfailed~U=Resource exception. unable to access the T due to limited resources~C=TSAFCACF~CKT=ONT:Pending_7_ONT,1V_VIDEO,1.1.1.1,,LET-8-5-1-1V,AFCM,ONT612,,SFU,,YES,[PENDING INSTALL],,,,,,,,:~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDS:ID=TELLABS-VIDEO-SIGNAL-CHECK~INS=~ST=FIOS-VODRF-INIT-VER1-STATE~CF=NETWORK~ST=1600238294~ED=1600238294~FT=PONEMS_FT~OS=PONEMS:RESSTART::RESS:N=VIDEO-SIGNAL-CHECK-INCOMPLETE-RESULT~I=~V=~EC=FT-EXCEPTION-SEND-FAILED~ED=PrimarySecondarybothfailed~U=Resource exception. Delphi unable to access the EMS/OLT due to limited resources~C=TSAFCACF~CKT=ONT:Pending_7_ONT,1V_VIDEO,1.1.1.1,,LET-8-5-1-1V,AFCM,ONT612,,SFU,,YES,[PENDING INSTALL],,,,,,,,:~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDS:ID=RETRIEVE-PON-ALARM~INS=RF~ST=FIOS-VODRF-INIT-VER1-STATE~-RESULT~I=RF~V=~EC=FAILED~ED=FAILED~U=FAILED~C=TSDELRVW~CKT=ONT:Pending_7_ONT,1V_VIDEO,1.1.1.1,,LET-8-5-1-1V,AFCM,ONT612,,SFU,,YES,[PENDING INSTALL],,,,,,,,:~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDS:ID=INSPECT-DATA-VIDEO-ASSOCIATION~INS=~ST=FIOS-VODRF-INIT-VER1-STATE~CF=NETWORK~ST=1600238294~ED=1600238296~FT=INVALID_FT~OS=:RESSTART::RESS:N=INSPECT-DATA-VIDEO-ASSOCIATION-OK-RESULT~I=~V=DATA-VIDEO-ARE-ASSOCIATED~EC=~ED=~U=Data and Video circuits are associated~C=TSDELTOK~CKT=BHR:,,,,,0,,,Y,0~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDEND:^ABOTHMO^^^ ^REAL^SYS_CMD_SVC_NOT_ELIGIBLE_TO_ARCHIVE^FAILED_TO_RETRIEVE_FROM_CACHE^0930^
@      
                                                      
09/16 07:08:18.17 rw/dte/(7323)3303 A245000004-^2020091676094516^76094516^DTRE^64/XXXX/62234    /VZXX^^^GET_GY:UTT^DATA^M^host1^07:08:17^07:08:18^1^1^:ANASTART::ANAS:RVW~Command failed to run, line record not found in iView, cannot process the command (8951)~TSDELCSD~Refresh. Manually troubleshoot service.~~~~~TSDELCSD:ANE::ANAEND:^2^:CMDSTART::CMDS:ID=MANAGE-LR-CACHE-COMMAND~INS=RETRIEVE~ST=FIOS-PATHINFO-CACHE-~C=TSDELCSD~CKT=~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDEND:^YANKOKA^^BDV_FORMAT,V2,Maintain individual node per BDV devices~INVOKE_ONT_UNLOCKED,NO,Invoke Unlocked ONT Serials in DTE~BRANDING,YES,Change from BHR to Router~^ ^^^^REAL^SYS_CMD_SVC_NOT_ELIGIBLE_TO_ARCHIVE^^0930^
      09/16 07:08:29.85 rw/dte/(7325)3303 A245000006-^2020091676094518^76094518^DTRE^11/KQXA/123/VXXX^^^GET_RR::UTT^DATA^M^HOST1^07:08:28^07:08:29^1^1^:ANASTART::ANAS:RVW~Command failed to run, line record not found in , cannot process the command (8951)~TSDELCSD~Refresh. Manually troubleshoot service.~~~~~TSDELCSD:ANE::ANAEND:^2^:CMDSTART::CMDS:ID=MANAGE-LR-CACHE-COMMAND~STATE~CF=NETWORK~ST=1600240108~ED=1600240109~D=NO VALID LINE RECORD FROM ,CPE~U=Line record not found in C=TSDELCSD~CKT=~C,Maintain individual node per BDV devices~INVOKE_ONT_UNLOCKED,NO,Invoke Unlocked ONT Serials in DTE~BRANDING,YES,Change from BHR to Router~^ ^^^^REAL^SYS_CMD_SVC_NOT_ELIGIBLE_TO_ARCHIVE^^0930^
      09/16 07:08:42.55 rw/dte/(7319)3303 A245000008-^2020091676094520^76094520^DTRE^110/KXXX/90876/VXXX^^^GET_TOPOLOGY::UTT^DATA^M^HOST1^07:08:41^07:08:42^1^1^:ANASTART::ANAS:RVW~Command failed to run, line record not found in iView, cannot process the command (8951)~TSDELCSD~Refresh. Manually troubleshoot service.~~~~~TSDELCSD:ANE::ANAEND:^2^:CMDSTART::CMDS:ID=MANAGE-LR-CACHE-COMMAND~INS=RETRIEVE~ST=FIOS-PATHINFO-CACHE-CF=NETWORK~ST=~RESSTART::RESS:N=-LR-FETCH-INCOMPLETE-RESULT~I=1~V=~EC=8951~ED=NO VALID LINE RECORD FROM ,CPE~U=Line record not found in iView~C=TSDELCSD~CKT=~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDEND:^YAN^^BDV_FORMAT,V2,Maintain individual node per BDV devices~INVOKE_ONT_UNLOCKED,NO,Invoke Unlocked ONT Serials in DTE~BRANDING,YES,Change from BHR to Router~^ ^^^^REAL^SYS_CMD_SVC_NOT_ELIGIBLE_TO_ARCHIVE^^0930^

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.