# Extract parameters from a message **URL:** \ **Category:** Logstash\ **Created:** [June 27, 2019, 9:32am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797 "2019-06-27T09:32:22Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![WarriorHarb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warriorharb/32/43201_2.png) [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Post date:** [June 27, 2019, 9:32am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/1 "2019-06-27T09:32:22Z")
Hello, So i have a message in my log that contains parameters that i need to extract. This is an log line exemple : > AgentID=50119&Tenant=2003&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=000000000 I need to extract Tenant value 50119 and put it in a variable Tenant. Now this would work fine if there was a space before Tenant, i would use this : ``` grok { break_on_match => false match => { "Message" => 'tenant (?[0-9]{4})'} } ``` But apparently since the words are not separated , this doesn't detect it. Any suggestions ? ---
**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\ **Post date:** [June 27, 2019, 9:44am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/2 "2019-06-27T09:44:10Z")
Hello @WarriorHarb, go to **Kibana - Dev Tools - Grok Debugger** **Sample data** ``` AgentID=50119&Tenant=2003&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=000000000 ``` **Grok pattern** ``` %{WARRIOR_GROK:debug} ``` **Custom patterns** ``` ALL_EXCEPT_AMP [^\&]* WARRIOR_GROK ^AgentID=%{ALL_EXCEPT_AMP}&Tenant=%{ALL_EXCEPT_AMP:tenant} ``` **Output** ``` { "debug": "AgentID=50119&Tenant=2003", "tenant": "2003" } ``` ---
**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\ **Post date:** [June 27, 2019, 9:49am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/3 "2019-06-27T09:49:35Z")
**GROK REGEXP** PUT GROK regexp to a file in directory /etc/logstash/grok-patterns ```auto #vim /etc/logstash/grok-patterns/warrior ALL_EXCEPT_AMP [^\&]* WARRIOR_GROK ^AgentID=%{ALL_EXCEPT_AMP}&Tenant=%{ALL_EXCEPT_AMP:tenant} ``` **FILTER GROK PLUGIN** In your logstash pipeline set the filter grok plugin. ``` grok { patterns_dir => ["/etc/logstash/grok-patterns"] break_on_match => false match => { "message" => ["%{WARRIOR_GROK}"] } overwrite => ["message"] tag_on_failure => ["parsing_failure_on_grok_warrior"] } ``` ---
**Author:** ![WarriorHarb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warriorharb/32/43201_2.png) [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Post date:** [June 27, 2019, 10:24am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/4 "2019-06-27T10:24:38Z")
Hello @vasek thanks for your quick response, The problem is that this parameter isn't always at the same place, > DialedNumber=000000000&ID=843&CountryCode=00&AreaCode=33& **Tenant=2002** &AgentID=0&GroupID=0&ScriptSource=1&AgentLeg=0 Also in some lines the Tenant doesn't exist at all ---
**Author:** ![BennyInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bennyinc/32/21751_2.png) [@BennyInc](https://discuss.elastic.co/u/BennyInc)\ **Post date:** [June 27, 2019, 10:35am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/5 "2019-06-27T10:35:27Z")
Have you considered using [the kv filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html) instead of the grok filter? If you are only interested in the tenant field, you could then use [the `include_keys` option](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html#plugins-filters-kv-include_keys) to only include that. ---
**Author:** ![WarriorHarb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warriorharb/32/43201_2.png) [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Post date:** [June 27, 2019, 10:41am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/6 "2019-06-27T10:41:12Z")
hi @BennyInc i got many other fields to detect with the same concept ---
**Author:** ![vasek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasek/32/136636_2.png) [@vasek](https://discuss.elastic.co/u/vasek)\ **Post date:** [June 27, 2019, 11:11am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/7 "2019-06-27T11:11:23Z")
You can try this: ``` ALL_EXCEPT_AMP [^\&]* WARRIOR_GROK (AgentID=%{ALL_EXCEPT_AMP:agent_id}|Tenant=%{ALL_EXCEPT_AMP:tenant}|GroupID=%{ALL_EXCEPT_AMP:groupd_id}|CountryCode=%{ALL_EXCEPT_AMP:country_code}|AreaCode=%{ALL_EXCEPT_AMP:area_code}|DialedNumber=%{ALL_EXCEPT_AMP:dialed_number}|&)* ``` It **doesn't matter on order** and **parameters can be empty** or **not present**. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 27, 2019, 1:56pm UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/8 "2019-06-27T13:56:26Z")
grok is not the right tool for this. Use a kv filter ``` kv { field_split => "&" include_keys => ["Tenant"] } ``` The include\_keys option is optional, by default it will extract every key/value pair. There is also an exclude\_keys option. ---
**Author:** ![WarriorHarb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warriorharb/32/43201_2.png) [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Post date:** [June 27, 2019, 11:44pm UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/9 "2019-06-27T23:44:33Z")
Hi @Badger, it worked fine when the tenant is in the middle but when it is in the beginning like this : ``` ` Tenant=2003&AgentID=50119&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=00000000` ``` it isn't detected at all. And in the case Tenant is in the end it detected everything in the rest of log line after the space like in this case : ``` AgentID=90610&AgentName=0ec3eb9e1159928d21e722996979d4e3&GroupID=19983&AgentLeg=1&DialedNumber=%2b33969780004&CountryCode=33&AreaCode=&Tenant=2002 Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5) ``` The Tenant value was : **2002 Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5)** that's right 2002 is the tenant ID but the rest is not correct. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 28, 2019, 12:13am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/10 "2019-06-28T00:13:09Z")
I would guess towards ``` kv { field_split => "&" trim_key => "` " } ``` but if you could provide an actual event and actual filter that does not work and an explanation of what is wrong with it that would help ---
**Author:** ![WarriorHarb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warriorharb/32/43201_2.png) [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Post date:** [June 28, 2019, 12:25am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/11 "2019-06-28T00:25:30Z")
This is i guess what you asked me for: an example of a log line : ``` 2019-05-25 01:21:03 W3SVC1 127.0.0.1 GET /DialPlan/FT_CheckPlan.aspx DialedNumber=%0000000000&Tenant=2138&ID=582&CountryCode=00&AreaCode=&AgentID=54567&GroupID=16707&ScriptSource=0&AgentLeg=1 Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5) ``` and another exemple witha different position of tenant : ``` 2019-05-25 01:22:49 W3SVC1 127.0.0.1 GET /DialPlan/FT_CheckPlan.aspx Tenant=2003&AgentID=50119&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=00000000 Mozilla/5.0+(Windows+NT;+Windows+NT+6.3;+en-US)+WindowsPowerShell ``` and Tenant could be at the end also followed by a space and then the rest of the log line (in this case the tenant value would be fetched but also the rest of the log line) After applying the grok filter and parsing the whole line into fields, i extract this whole part in a variable messageA : ``` Tenant=2003&AgentID=50119&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=00000000 ``` then i would like to extract the Tenant value. I applied as u suggested the KV filter : > kv { field\_split =\> "&" include\_keys =\> ["Tenant"] } it extracted the tenant ID when it was located in the middle, but when it's in the beginning ,the ID isn't detected. ---
**Author:** ![BennyInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bennyinc/32/21751_2.png) [@BennyInc](https://discuss.elastic.co/u/BennyInc)\ **Post date:** [June 28, 2019, 9:13am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/12 "2019-06-28T09:13:49Z")
> [@WarriorHarb](#): > > I applied as u suggested the KV filter : > > > kv { field\_split =\> "&" include\_keys =\> ["Tenant"] } > > it extracted the tenant ID when it was located in the middle, but when it's in the beginning ,the ID isn't detected. You would need to make sure the kv filter uses the correct source field: ```auto kv { field_split => "&" include_keys => ["Tenant"] source => "messageA" } ``` [By default it reads from the `message` field.](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html#plugins-filters-kv-source) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 26, 2019, 9:13am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797/13 "2019-07-26T09:13:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.