Parsing log with comma separated key value pairs

[Abhilash_B]

Hi all,

I have a log file some like this.

2019-01-30T06:01:02.551-07:00;82828;10400035;INFO;Measure    vehicle detected;System     Vehicle detected;"CAL_CHKS=,CAL_CPPX=0,CAL_CPPY=0,CAM_CUSR=CAM_CUSR,CAM_NUMB=1,CAM_NUS2=CAM_NUS2,CAM_NUS3=CAM_NUS3,CAM_NUS4=CAM_NUS4,CAM_NUS5=CAM_NUS5,CAM_NUS6=CAM_NUS6,CAM_NUSR=CAM_NUSR,CAM_NWID=0030D60B3825,CAM_NWIP=10.241.155.73,CAM_SERN=625-011/60815,CAM_SYSN=,CAM_VERS=4.4.1,KDK29.0D_C,NONE,CFG_ATYP=4,CFG_AZIM=22.0,CFG_CAMH=0.00,CFG_CANX=90.00,CFG_CANY=0.00,CFG_CANZ=-68.00,CFG_CPOX=0.00,CFG_CPOY=0.00,CFG_DMOD=0,CFG_ENCR=0,CFG_FSKS=2,CFG_IPP1=1,CFG_L1WI=3200,CFG_L2WI=3199,CFG_L3WI=0,CFG_L4WI=0,CFG_L5WI=0,CFG_L6WI=0,CFG_LANS=0,CFG_LENS=0,CFG_LLEN=0000,CFG_LOFF=3200,CFG_MAXR=0,CFG_MINA=1.4,CFG_NIMG=1,CFG_NUML=2,CFG_PHOD=L,CFG_RDEL=0.1,CFG_RESX=0.00,CFG_RESY=0.00,CFG_SIGN=1,CFG_SLNG=--,CFG_STYP=S,CFG_TPOS=3,CFG_WVZM=0,CLK_SOUR=manual,CLK_STAT=1,DEV_CHKS=25F50000,DEV_SERN=#001722AC6001,DEV_SYSN=ISK,DEV_VERS=ISK_RUSPE79,ESM_FRMT=nodevice,ESM_LABL=-,ESM_MANU=-,ESM_PROD=-,ESM_SERN=-,ESM_SIZE=0,ESM_TYPE=-,FOT_POSX=30.12,FOT_POSY=4.40,FRA_TYPE=M,INC_CREE=010,INC_CVEH=0,INC_DATE=300119,INC_EVID=097,INC_LANC=3,INC_LANE=1,INC_LIMI=050,INC_MLIM=061,INC_MMOD=0,INC_MONT=January,INC_NIMG=0,INC_PHOP=F,INC_QUAL=9,INC_SFMT=0,INC_SPEE=044,INC_SPEP=---,INC_SPWR= ,INC_TIME=060102,INC_TLIG=1,INC_TLPH=-,INC_TLST=0,INC_TPOS=-1,INC_TYPE=,INC_TZON=MST,INC_VEHT=Car,ISO_DATE=2019-01-30,ISO_LIMI=081,ISO_MLIM=099,ISO_SPEE=072,LGY_FIDX=0,LOC_LI_L=050,LOC_LI_P=050,LOC_TEX0=ME80,LOC_TEX1=NB S ELLSWORTH RD @ E PECOS RD,LOC_TEX2=AZMES,LOC_TEX3=,LOC_TEX4=,LOC_TEX5=LID,LOC_TEX6=NOCLS,LOC_TEX7=,LOC_TEX8=,LOC_TEX9=,LOG_USER=admin,MEA_CHKS=DE4549FB,MEA_NUMB=0,MEA_RANG=0,MEA_SYSN=TraffiStar SR390,MEA_UNIT=mph ,MEA_VERS=SR390.SC41.T.18102512,PRO_KIND= ,RRS_LTHR=6,RRS_OTHR=2,RRS_TYPE=2,RRS_VLEN=400,SCR__POS=TOP,SEN_CHKS=788A0186,SEN_LOOP=003000062030000322,103000096030000662,200000000000000002,300000000000000002,SEN_POSX=29.90,SEN_POSY=4.40,SEN_RSIT=0000000000540220-088,SEN_RZON=000000000000,SEN_SERN=590-112/62328,SEN_SYSN=FS3,SEN_VERS=24F_ST_3G1J,SSV_DATA=,SW__CHKS=C0FB24A5,SYS_BPLV=G93,SYS_CHKS=08DA1FCA,SYS_IDNT=0030D60B3825,TRI_LI_L=099,TRI_LI_P=099,ZUL_CHKS=C0FB24A5,ZUL_KATE=,ZUL_STRI=,ZUL_TEX3=,ZUL_TEX4=,ZUL_TEX5=,ZUL_TEX6="

The main event is separated by ";" and the last part of the main event are further comma separated key value pairs. I am confused about what filters to be used for this usecase. Any help is appreciated.

[OlegInishev]

Maybe you need GROK and KV filter

%{TIMESTAMP_ISO8601};%{INT:field1};%{INT:field2};%{WORD:type_message};%{DATA:Data1};%{DATA:Data2};%{QS:For_KV_filter}

[Badger]

Use grok to break it into three parts. You could use GREEDYDATA to capture middleBit, but I have an aversion for using GREEDYDATA anywhere except in last place. Then you can use dissect or csv for the middle bit, and kv for the end.

    grok { match => { "message" => '^(?<ts>[^;]+);(?<middleBit>[^"]+);"%{DATA:restOfLine}"' } }
    dissect { mapping => { "middleBit" => "%{[middle][a]};%{[middle][b]};%{[middle][c]};%{[middle][d]};%{[middle][e]}" } }
    kv { source => "restOfLine" target => "keys" field_split => "," }