Hi,
I have my log in below format, it actually has all fields delimited by ^. However, in between we can have few other lines which are nothing but the field headers. I am unable to get this log converted into json format using logstash like below:
Expected Json Format:
{"SESSIONID":"A335000000-","TRANID":"2020091676094508","WPID":"76094508","SYSNAME ":"VRP" and so on ....}
Can anyone help me hot to achieve this as it is very much needed for me.
System Log Format:
-=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=- -=*=-
@ 09/16 06:38:16.46 @@start@@
$
09/16 06:38:16.46 rw/dte/(24538)3218
01 SESSIONID 02 TRANID 03 WPID 04 SYSNAME 05 CID 06 TN
07 ORDERNO 08 COMMAND 09 SVCTYPE 10 WKTYPE 11 HOST 12 IN
13 OUT 14 DIFF 15 NOOF ANA 16 ANALYSIS 17 NOOF CMD 18 COMMANDS
19 EMPID 20 NID 21 ATTRIBUTES 22 CACHE 23 CKTMODEL 24 TSYS
25 TSTMODE 26 CACH ARCH R 27 CACH RTRV R 28 RETC
16 ANALYSIS EXPANSION
01 DC 02 EXP 03 TRBLCODE 04 ACTION 05 STB SER 06 NETYPE
07 NEID
17 COMMAND EXPANSION
01 ID 02 INSTANCE 03 STATENAME 04 CMDFMLY 05 START 06 END
17 COMMAND RESULT EXPANSION
01 NAME 02 INSTANCE 03 INTVAL 04 ERRCODE 05 ERRMSG 06 USERFVAL
07 TRBLCODE 08 NESTR 09 COMMENT 10 EXP
09/16 06:38:16.47 rw/dte/(24538)3303 A335000000-^2020091676094508^76094508^VRP^22/VXXA/12345 /VZMA^^^QT::ASO^VIDEO^M^dhost1^06:36:58^06:38:16^78^1^:ANASTART::ANAS:RVW~Systemunable to communicate with Tellabs Element Management System. Access Guardian Error: Unable to open connection. Please retry this request later.~TSAFCACF~Retry, follow M&P~~ONT~[PENDING INSTALL]~~:ANE::ANAEND:^15^:CMDSTART::CMDS:ID=MANAGE-LR-CACHE-COMMAND~INS=RETRIEVE~ST=FIOS-PATHINFO-CACHE-STATE~CF=NETWORK~ST=1600238218~ED=1600238218~FT=CACHE_FT~OS=CACHESRV:RESSTART::RESEND::CMDE::CMDS:ID=GET-CSD2~INS=1~ST=FIOS-PATHINFO-INIT-STATE~CF=NETWORK~ST=1600238218~ED=1600238256~FT=IVIEW_FT~OS=IVIEW_FT:RESSTART::RESEND::CMDE::CMDS:ID=GET~INS=7812516069~ST=FIOS-PATHINFO-STATEA~CF=NETWORK~ST=1600238256~ED=1600238279~RESSTART::RESEND::CMDE::CMDS:ID=GET-CSD2~INS=7813205883~ST=FIOS-PATHINFO-STATEA~CF=NETWORK~ST=1600238256~ED=1600238279~FT=IVIEW_FT~OS=RESEND::CMDE::CMDS:ID=GET-er::CMDE::CMDS:ID=FIOS-FINAL-COMMAND-PATHINFO~INS=~ST=FIOS-PATHINFO-END-STATE~CF=NETWORK~ST=1600238294~ED=1600238294~FT=INVALID_FT~OS=:RESSTART::RESEND::CMDE::CMDS:ID=RETRIEVE-TEST-HISTORY-COMMAND~INS=CACHE_RESULTS~ST=FIOS-RETRIEVE-CACHE-RESULTS-STATE-A~CF=RT::RESS:N=CACHE-HISTORY-OK-RESULT~I=CACHE_RESULTS~V=NO-RESULTS~EC=~ED=~U=Retrieve Cache History Command Successful~C=TSDELTOK~CKT=~CMT=1~EXP=0:RESE::RESEND::CMDE::CMDS:ID=RTRV-FT-RESP-COMMAND~INS=RTRV-FT-RESP~ST=FIOS-RETRIEVE-CACHE-RESULTS-STATE-B~CF=NETWORK~ST=~ED=~FT=IGNITE_FT~OS=::RESEND::CMDE::CMDS:ID=RETRV-IGNITE-GENERIC-KEY-EXECUTION-STATUS-COMMAND~INS=RETRV~ST=FIOS-RETRIEVE-INPROGRESS-IG-STATE-C~CF=NETWORK~ST=1600238294~ED=1600238294~FT=IGNITE_FT~OS=IGNITE:RESSTART::RESEND::CMDE::CMDS:ID=RETRIEVE-PON-ALARM~INS=~ST=FIOS-VODRF-INIT-VER1-STATE~CF=NETWORK~ST=1600238294~ED=1600238294~FT=PONNMS_FT~OS=WSG_PONNMS:RESSTART::RESS:N=OLT-ALARMS-INCOMPLETE-RESULT~I=~V=~EC=FAILED~ED=FAILED~U=FAILED~C=TSDELRVW~CKT=ONT:Pending_7_ONT,X_1V_DATA,1.1.1,LET-8-5-1-1V,AFCM,ONT612,,SFU,,YES,[PENDING INSTALL],,,25 Mbps Mpbs,25 Mbps,25 Mbps,25 Mbps Mpbs,25 Mbps,25 Mbps:~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDS:ID=TELLABS-VIDEO-PORT-STATUS~INS=~ST=FIOS-VODRF-INIT-VER1-STATE~CF=NETWORK~ST=1600238294~ED=1600238294~FT=PONEMS_FT~OS=PONEMS:RESSTART::RESS:N=VIDEO-PORT-STATUS-INCOMPLETE-RESULT~I=~V=~EC=FT-EXCEPTION-SEND-FAILED~ED=PrimarySecondarybothfailed~U=Resource exception. unable to access the T due to limited resources~C=TSAFCACF~CKT=ONT:Pending_7_ONT,1V_VIDEO,1.1.1.1,,LET-8-5-1-1V,AFCM,ONT612,,SFU,,YES,[PENDING INSTALL],,,,,,,,:~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDS:ID=TELLABS-VIDEO-SIGNAL-CHECK~INS=~ST=FIOS-VODRF-INIT-VER1-STATE~CF=NETWORK~ST=1600238294~ED=1600238294~FT=PONEMS_FT~OS=PONEMS:RESSTART::RESS:N=VIDEO-SIGNAL-CHECK-INCOMPLETE-RESULT~I=~V=~EC=FT-EXCEPTION-SEND-FAILED~ED=PrimarySecondarybothfailed~U=Resource exception. Delphi unable to access the EMS/OLT due to limited resources~C=TSAFCACF~CKT=ONT:Pending_7_ONT,1V_VIDEO,1.1.1.1,,LET-8-5-1-1V,AFCM,ONT612,,SFU,,YES,[PENDING INSTALL],,,,,,,,:~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDS:ID=RETRIEVE-PON-ALARM~INS=RF~ST=FIOS-VODRF-INIT-VER1-STATE~-RESULT~I=RF~V=~EC=FAILED~ED=FAILED~U=FAILED~C=TSDELRVW~CKT=ONT:Pending_7_ONT,1V_VIDEO,1.1.1.1,,LET-8-5-1-1V,AFCM,ONT612,,SFU,,YES,[PENDING INSTALL],,,,,,,,:~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDS:ID=INSPECT-DATA-VIDEO-ASSOCIATION~INS=~ST=FIOS-VODRF-INIT-VER1-STATE~CF=NETWORK~ST=1600238294~ED=1600238296~FT=INVALID_FT~OS=:RESSTART::RESS:N=INSPECT-DATA-VIDEO-ASSOCIATION-OK-RESULT~I=~V=DATA-VIDEO-ARE-ASSOCIATED~EC=~ED=~U=Data and Video circuits are associated~C=TSDELTOK~CKT=BHR:,,,,,0,,,Y,0~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDEND:^ABOTHMO^^^ ^REAL^SYS_CMD_SVC_NOT_ELIGIBLE_TO_ARCHIVE^FAILED_TO_RETRIEVE_FROM_CACHE^0930^
@
09/16 07:08:18.17 rw/dte/(7323)3303 A245000004-^2020091676094516^76094516^DTRE^64/XXXX/62234 /VZXX^^^GET_GY:UTT^DATA^M^host1^07:08:17^07:08:18^1^1^:ANASTART::ANAS:RVW~Command failed to run, line record not found in iView, cannot process the command (8951)~TSDELCSD~Refresh. Manually troubleshoot service.~~~~~TSDELCSD:ANE::ANAEND:^2^:CMDSTART::CMDS:ID=MANAGE-LR-CACHE-COMMAND~INS=RETRIEVE~ST=FIOS-PATHINFO-CACHE-~C=TSDELCSD~CKT=~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDEND:^YANKOKA^^BDV_FORMAT,V2,Maintain individual node per BDV devices~INVOKE_ONT_UNLOCKED,NO,Invoke Unlocked ONT Serials in DTE~BRANDING,YES,Change from BHR to Router~^ ^^^^REAL^SYS_CMD_SVC_NOT_ELIGIBLE_TO_ARCHIVE^^0930^
09/16 07:08:29.85 rw/dte/(7325)3303 A245000006-^2020091676094518^76094518^DTRE^11/KQXA/123/VXXX^^^GET_RR::UTT^DATA^M^HOST1^07:08:28^07:08:29^1^1^:ANASTART::ANAS:RVW~Command failed to run, line record not found in , cannot process the command (8951)~TSDELCSD~Refresh. Manually troubleshoot service.~~~~~TSDELCSD:ANE::ANAEND:^2^:CMDSTART::CMDS:ID=MANAGE-LR-CACHE-COMMAND~STATE~CF=NETWORK~ST=1600240108~ED=1600240109~D=NO VALID LINE RECORD FROM ,CPE~U=Line record not found in C=TSDELCSD~CKT=~C,Maintain individual node per BDV devices~INVOKE_ONT_UNLOCKED,NO,Invoke Unlocked ONT Serials in DTE~BRANDING,YES,Change from BHR to Router~^ ^^^^REAL^SYS_CMD_SVC_NOT_ELIGIBLE_TO_ARCHIVE^^0930^
09/16 07:08:42.55 rw/dte/(7319)3303 A245000008-^2020091676094520^76094520^DTRE^110/KXXX/90876/VXXX^^^GET_TOPOLOGY::UTT^DATA^M^HOST1^07:08:41^07:08:42^1^1^:ANASTART::ANAS:RVW~Command failed to run, line record not found in iView, cannot process the command (8951)~TSDELCSD~Refresh. Manually troubleshoot service.~~~~~TSDELCSD:ANE::ANAEND:^2^:CMDSTART::CMDS:ID=MANAGE-LR-CACHE-COMMAND~INS=RETRIEVE~ST=FIOS-PATHINFO-CACHE-CF=NETWORK~ST=~RESSTART::RESS:N=-LR-FETCH-INCOMPLETE-RESULT~I=1~V=~EC=8951~ED=NO VALID LINE RECORD FROM ,CPE~U=Line record not found in iView~C=TSDELCSD~CKT=~CMT=0~EXP=0:RESE::RESEND::CMDE::CMDEND:^YAN^^BDV_FORMAT,V2,Maintain individual node per BDV devices~INVOKE_ONT_UNLOCKED,NO,Invoke Unlocked ONT Serials in DTE~BRANDING,YES,Change from BHR to Router~^ ^^^^REAL^SYS_CMD_SVC_NOT_ELIGIBLE_TO_ARCHIVE^^0930^