# Grok to extract info from non KV formatted data? **URL:** \ **Category:** Logstash\ **Created:** [September 3, 2019, 4:12pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901 "2019-09-03T16:12:38Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![SigmazGFX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sigmazgfx/32/53534_2.png) [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Post date:** [September 3, 2019, 4:12pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901/1 "2019-09-03T16:12:38Z")
Ok.. I'm back with another head scratcher.. I'm so damn new... but I'm learning.😃 I'm sending my entire message through kv{} to have it break out the KV pairs for me.. but I need to make a KV pair for items listed that aren't logged with "this=that" For instance, the value I want to harvest is from our ModSecurity log entry.. I would like to be able to analyze the triggered rule ID's i.e. [id "981203"] What Grok would I use to have logstash seek out and create a KV pair for this bit of info? Something like owasp\_rule = "981203" or whatever it should be.. The kv{} filter is doing a great job on the this="that" type data but I need to be able to pick the informational stuff out too.. like id, msg, line, rev, accuracy... The data coming into Logstash looks like this: [Fri Aug 30 09:19:31.035231 2019] [security2:error] [pid 3064:tid 140334373766912] [client x.x.x.x:45747] [client x.x.x.x] ModSecurity: Warning. String match "bytes=0-" at REQUEST\_HEADERS:Range. [file "/content/waf/2.7.3/modsecurity\_crs\_protocol\_violations.conf"] [line "427"] [id "958291"] [rev "2"] [msg "Range: field exists and begins with 0."] [data "bytes=0-"] [severity "WARNING"] [ver "OWASP\_CRS/2.2.7"] [maturity "6"] [accuracy "8"] [tag "OWASP\_CRS/PROTOCOL\_VIOLATION/INVALID\_HREQ"] [hostname "[server.domain.com](http://server.domain.com)"] [uri "/path-to-resource"] [unique\_id "XWkiY38AAAEAAAv4@34AAAAM"], referer: [https://server.domain.com/default.aspx](https://server.domain.com/default.aspx) [Fri Aug 30 09:19:31.016044 2019] timestamp="1567171171" srcip="x.x.x.x" localip="y.y.y.y" user="-" host="z.z.z.z" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" duration="30788" url="/default.aspx" server="[server.domain.com](http://server.domain.com)" referer="[https://server.domain.com/default.aspx](https://server.domain.com/default.aspx)" cookie="ASP.NET\_SessionId=tazkoispw5xrymn2vjz5uung; SecurityToken=SecurityTokenID=4ds3545-n34-4hfc-2dhy-4kgju76s458c&Issued=8/30/2019 1:18:38 PM&Expires=8/30/2019 11:18:38 PM; UserInfo=UserID=0&UserID=0&TAID=0&Username=jSchmoe&FullName=Joe [Schmoe&Email=Jschmoe@no.net](mailto:Schmoe&Email=Jschmoe@no.net)&Phone=&Extension=" set-cookie="-" recvbytes="957" sentbytes="17206" protocol="HTTP/1.1" ctype="text/html" uagent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36" querystring="" websocket\_scheme="-" websocket\_protocol="-" websocket\_key="-" websocket\_version="-" ruleid="31" Thanks in advance for any insight you guys can offer a newb. ---
**Author:** ![SigmazGFX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sigmazgfx/32/53534_2.png) [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Post date:** [September 4, 2019, 12:20pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901/2 "2019-09-04T12:20:11Z")
Anyone? Please?.. lol ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 4, 2019, 1:01pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901/3 "2019-09-04T13:01:42Z")
You need to strip off the parts that do not match but a kv filter can handle that ModSecurity line ``` mutate { gsub => ["message", ".*ModSecurity: [^\[]+\[", ""] } mutate { gsub => ["message", "][^\[]+$", "" ] } kv { field_split_pattern => "] \[" value_split => " " } ``` will get you ``` "tag" => "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ", "rev" => "2", "msg" => "Range: field exists and begins with 0.", "id" => "958291", "unique_id" => "XWkiY38AAAEAAAv4@34AAAAM", "line" => "427", "ver" => "OWASP_CRS/2.2.7", "data" => "bytes=0-", "severity" => "WARNING", "maturity" => "6", "hostname" => "server.domain.com", "file" => "/content/waf/2.7.3/modsecurity_crs_protocol_violations.conf", "uri" => "/path-to-resource" ``` ---
**Author:** ![SigmazGFX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sigmazgfx/32/53534_2.png) [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Post date:** [September 4, 2019, 1:07pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901/4 "2019-09-04T13:07:02Z")
Badger.. you have been really helpful.. Thanks for offering me the info.. I do appreciate it.. So would i add a secodary instance of the KV filter in my .conf? or do I inline this new tranformation with the other one you suggested.. Which is working great thanks! filter { mutate { gsub =\> ["message", ".\*ModSecurity: [^+[", ""] } mutate { gsub =\> ["message", "][^+$", ""] } kv { field\_split\_pattern =\> "] [" value\_split =\> " " } dissect { mapping =\> { "message" =\> "[%{[@metadata][timestamp]}]%{}" } } date { match =\> ["[@metadata][timestamp]", "EEE MMM dd HH:mm:ss.SSSSSS yyyy" ] } } ---
**Author:** ![SigmazGFX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sigmazgfx/32/53534_2.png) [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Post date:** [September 4, 2019, 1:08pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901/5 "2019-09-04T13:08:40Z")
or this would be a argument for an IF statement.. because the ModSecurity info isn't always there. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 4, 2019, 1:21pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901/6 "2019-09-04T13:21:46Z")
Yes, you should use if so that you only apply the kv filter to lines that you expect to match it. ---
**Author:** ![SigmazGFX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sigmazgfx/32/53534_2.png) [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Post date:** [September 4, 2019, 1:23pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901/7 "2019-09-04T13:23:53Z")
Ok. Sounds good, I'll try to cobble something together today. You've been a great help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 2, 2019, 1:23pm UTC](https://discuss.elastic.co/t/grok-to-extract-info-from-non-kv-formatted-data/197901/8 "2019-10-02T13:23:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.