# Strange behaviour kv field\_split **URL:** \ **Category:** Logstash\ **Created:** [February 13, 2020, 3:20pm UTC](https://discuss.elastic.co/t/strange-behaviour-kv-field-split/219234 "2020-02-13T15:20:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Fred3](https://avatars.discourse-cdn.com/v4/letter/f/f07891/32.png) [@Fred3](https://discuss.elastic.co/u/Fred3)\ **Post date:** [February 13, 2020, 3:20pm UTC](https://discuss.elastic.co/t/strange-behaviour-kv-field-split/219234/1 "2020-02-13T15:20:20Z")
Hi All, I'm trying to parse a log with the character "|" as split field value. It works fine for all fields but I get extra fields which are really broken and consists of multiple field-value-pairs, e.g. - 01-01015173|url=http.keyword - BP 437 "Wohnen am Klucksgraben - Erschließung EFH-Gebiet|url=https.keyword - AIRF)|pComp=CAV|riskType=1|srcZone=0dstZone=1|detectionType=1|act=not blocked|threatType=1|hostName=www.proxylists.net|cnt=1|aggregatedCnt=1|cccaDestinationFormat=URL|cccaDetectionSource=GLOBAL\_INTELLIGENCE|cccaRiskLevel=2|cccaDestination=http My config is as follows: > ``` > filter > { > if [type] == "test1" > { > mutate { > gsub => ["message", "\t", "|"] > } > kv { > source => "message" > value_split => "=" > field_split => "|" > } > } > } > > ``` I also found warnings in logstash logs that says: `[WARN][logstash.filters.kv][main] Exception while parsing KV {:exception=>"Invalid FieldReference: ` THO@UNER.DE|suser=UC@UNER.DE|mailMsgSubject=[TURM] test|url=https`"}` Could you help me please? ---
**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\ **Post date:** [February 17, 2020, 10:28am UTC](https://discuss.elastic.co/t/strange-behaviour-kv-field-split/219234/2 "2020-02-17T10:28:33Z")
Hi Could you please post a sample of your input as seen by the filter? On way to do it is deactivate the filter and post the ouptut of `stdout{}`. ---
**Author:** ![Fred3](https://avatars.discourse-cdn.com/v4/letter/f/f07891/32.png) [@Fred3](https://discuss.elastic.co/u/Fred3)\ **Post date:** [February 17, 2020, 11:41am UTC](https://discuss.elastic.co/t/strange-behaviour-kv-field-split/219234/3 "2020-02-17T11:41:11Z")
Hello again, here are some samples. I deactivated the filter like you described it. (I edited sensitive information into useless strings but kept the syntax). ``` <158>LEEF:1.0|Trend Micro|Deep Discovery Inspector|5.5.1200|SECURITY_RISK_DETECTION|devTimeFormat=MMM dd yyyy HH:mm:ss z ptype=IDS dvc=111.111.111.111 deviceMacAddress=11:11:11:11:11:11 dvchost=NAME1 deviceGUID=1A1A1A1A1A1A-1A1A1A1A-1A1A-1A1A-1A1A devTime=Feb 17 2020 12:21:41 GMT+01:00 sev=2 protoGroup=SQL proto=MSSQL vLANId=1111 deviceDirection=1 dhost=name-name-1111 dst=111.111.111.111 dstPort=1114 dstMAC=11:11:11:11:11:11 shost=name.domain.de src=111.111.111.111 srcPort=1113 srcMAC=11:11:11:11:11:11 malType=OTHERS fileType=-65536 fsize=0 ruleId=504 msg=Successful log on to service deviceRiskConfidenceLevel=3 pComp=NCIE riskType=1 srcGroup=Name Network srcZone=1 dstZone=0 detectionType=1 act=not blocked threatType=2 interestedIp=111.111.111.111 peerIp=111.111.111.111 dUser1=domain\h1111 cnt=1 aggregatedCnt=1 evtCat=Authentication evtSubCat=Login Success aptRelated=0 pAttackPhase=Asset and Data Discovery <158>LEEF:1.0|Trend Micro|Deep Discovery Inspector|5.5.1200|SECURITY_RISK_DETECTION|devTimeFormat=MMM dd yyyy HH:mm:ss z ptype=IDS dvc=111.111.111.111 deviceMacAddress=11:11:11:11:11:11 dvchost=NAME1 deviceGUID=1A1A1A1A1A1A-1A1A1A1A-1A1A-1A1A-1A1A devTime=Feb 17 2020 12:02:07 GMT+01:00 sev=2 protoGroup=CIFS proto=SMB2 vLANId=1115 deviceDirection=0 dhost=name.domain.de dst=111.111.111.111 dstPort=145 dstMAC=11:11:11:11:11:11 shost=111.111.111.111 src=111.111.111.111 srcPort=1113 srcMAC=11:11:11:11:11:11 malType=OTHERS filePath=H1111\Outlook\ fname=~archive.pst.tmp fileType=327680 fsize=4096 ruleId=4222 msg=PST File Upload deviceRiskConfidenceLevel=3 pComp=NCIE riskType=1 srcZone=0 dstGroup=Name Network dstZone=1 detectionType=1 act=not blocked threatType=2 interestedIp=111.111.111.111 peerIp=111.111.111.111 fileHash=9296DE94851FF12571C42E80D78D169AAC683CD3 sUser1=domain\h1111 cnt=3 aggregatedCnt=1 evtCat=File aptRelated=0 pAttackPhase=Data Exfiltration <158>LEEF:1.0|Trend Micro|Deep Discovery Inspector|5.5.1200|SECURITY_RISK_DETECTION|devTimeFormat=MMM dd yyyy HH:mm:ss z ptype=IDS dvc=111.111.111.111 deviceMacAddress=11:11:11:11:11:11 dvchost=NAME1 deviceGUID=1A1A1A1A1A1A-1A1A1A1A-1A1A-1A1A-1A1A devTime=Feb 17 2020 12:19:02 GMT+01:00 sev=6 protoGroup=SMTP proto=SMTP vLANId=4095 deviceDirection=1 dhost=111.111.111.111 dst=111.111.111.111 dstPort=111 dstMAC=11:11:11:11:11:11 shost=name.domain.intern src=111.111.111.111 srcPort=1117 srcMAC=11:11:11:11:11:11 malType=OTHERS fileType=393871360 fsize=1029 ruleId=29 msg=Unregistered sender and recipient domains - Email deviceRiskConfidenceLevel=3 duser=name@domain.de suser=name@domain.de mailMsgSubject=Reigabe Vong 180009: Prg und Gigung move eltor G ( 0102 ) pComp=CAV riskType=1 srcGroup=Name Network srcZone=1 dstZone=0 detectionType=1 act=not blocked threatType=2 interestedIp=111.111.111.111 peerIp=111.111.111.111 fileHash=5A289B28850A26C12882EC11AD45E5D78C16775D cnt=1 aggregatedCnt=1 evtCat=Suspicious Traffic evtSubCat=Email aptRelated=0 ``` ---
**Author:** ![Fred3](https://avatars.discourse-cdn.com/v4/letter/f/f07891/32.png) [@Fred3](https://discuss.elastic.co/u/Fred3)\ **Post date:** [February 19, 2020, 12:14pm UTC](https://discuss.elastic.co/t/strange-behaviour-kv-field-split/219234/4 "2020-02-19T12:14:18Z")
Update: The failure for this index disappeared. I checked today and all fields are parsed correctly. But I got the same strange error after editing another index. For me it seems that the error occures than a (broken) field is created I didn't defined. Here is an example which is related to the config and log I already posted: Logstash is creating the message field and all fields located in it correctly, e.g.: ``` message = <158>LEEF:1.0|Trend Micro|Deep Discovery Inspector|5.5.1200|SECURITY_RISK_DETECTION|devTimeFormat=MMM dd yyyy HH:mm:ss z ptype=IDS dvc=111.111.111.111 deviceMacAddress=11:11:11:11:11:11 dvchost=NAME1 deviceGUID=1A1A1A1A1A1A-1A1A1A1A-1A1A-1A1A-1A1A devTime=Feb 17 2020 12:21:41 GMT+01:00 sev=2 protoGroup=SQL proto=MSSQL vLANId=1111 deviceDirection=1 dhost=name-name-1111 dst=111.111.111.111 dstPort=1114 dstMAC=11:11:11:11:11:11 shost=name.domain.de src=111.111.111.111 srcPort=1113 srcMAC=11:11:11:11:11:11 malType=OTHERS fileType=-65536 fsize=0 ruleId=504 msg=Successful log on to service deviceRiskConfidenceLevel=3 pComp=NCIE riskType=1 srcGroup=Name Network srcZone=1 dstZone=0 detectionType=1 act=not blocked threatType=2 interestedIp=111.111.111.111 peerIp=111.111.111.111 dUser1=domain\h1111 cnt=1 aggregatedCnt=1 evtCat=Authentication evtSubCat=Login Success aptRelated=0 pAttackPhase=Asset and Data Discovery pComp = Sandbox tags = Deep Discovery Inspector ``` But the additional field **\<158\>LEEF** is created which I didn't defined: ``` <158>LEEF = 1.0|Trend Micro|Deep Discovery Inspector|5.5.1200|SECURITY_RISK_DETECTION|devTimeFormat=MMM dd yyyy HH:mm:ss z ptype=IDS dvc=111.111.111.111 deviceMacAddress=11:11:11:11:11:11 dvchost=NAME1 deviceGUID=1A1A1A1A1A1A-1A1A1A1A-1A1A-1A1A-1A1A devTime=Feb 17 2020 12:21:41 GMT+01:00 sev=2 protoGroup=SQL proto=MSSQL vLANId=1111 deviceDirection=1 dhost=name-name-1111 dst=111.111.111.111 dstPort=1114 dstMAC=11:11:11:11:11:11 shost=name.domain.de src=111.111.111.111 srcPort=1113 srcMAC=11:11:11:11:11:11 malType=OTHERS fileType=-65536 fsize=0 ruleId=504 msg=Successful log on to service deviceRiskConfidenceLevel=3 pComp=NCIE riskType=1 srcGroup=Name Network srcZone=1 dstZone=0 detectionType=1 act=not blocked threatType=2 interestedIp=111.111.111.111 peerIp=111.111.111.111 dUser1=domain\h1111 cnt=1 aggregatedCnt=1 evtCat=Authentication evtSubCat=Login Success aptRelated=0 pAttackPhase=Asset and Data Discovery ``` After I edit an index and restart logstash to activate my changes the field **\<158\>LEEF** breaks so that additional fields are created which contain the remaining parts of the event. Here is an example: ``` <158>LEEF = 1.0|Trend Micro|Deep Discovery Inspector|5.5. 1200|SECURITY_RISK_DETECTION|devTimeFormat=MMM = dd yyyy HH:mm:ss z ptype=IDS dvc=111.111.111.111 deviceMacAddress=11:11:11:11:11:11 dvchost=NAME1 deviceGUID=1A1A1A1A1A1A-1A1A1A1A-1A1A-1A1A-1A1A devTime=Feb 17 2020 12:21:41 GMT+01:00 sev=2 protoGroup=SQL proto=MSSQL vLANId=1111 deviceDirection=1 dhost=name-name-1111 dst=111.111.111.111 dstPort=1114 dstMAC=11:11:11:11:11:11 shost=name.domain.de src=111.111.111.111 srcPort=1113 srcMAC=11:11:11:11:11:11 malType=OTHERS fileType=-65536 fsize=0 ruleId=504 msg=Successful log on to service deviceRiskConfidenceLevel=3 pComp=N ``` Do you have an idea why the field **\<158\>LEEF** is created and what causes the split of it? ---
**Author:** ![Fred3](https://avatars.discourse-cdn.com/v4/letter/f/f07891/32.png) [@Fred3](https://discuss.elastic.co/u/Fred3)\ **Post date:** [March 11, 2020, 4:21pm UTC](https://discuss.elastic.co/t/strange-behaviour-kv-field-split/219234/5 "2020-03-11T16:21:09Z")
After checking all configuration files I found out that I missed a check for the type in one of them. This caused Logstash to parse events from other sources in the wrong index. I fixed it and now everything is working fine. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 8, 2020, 4:21pm UTC](https://discuss.elastic.co/t/strange-behaviour-kv-field-split/219234/6 "2020-04-08T16:21:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.