Extract parameters from a message

Elastic Stack Logstash
1

Hello,
So i have a message in my log that contains parameters that i need to extract.
This is an log line exemple :

AgentID=50119&Tenant=2003&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=000000000

I need to extract Tenant value 50119 and put it in a variable Tenant.
Now this would work fine if there was a space before Tenant, i would use this :

			grok {
			break_on_match => false
			match => { "Message" => 'tenant (?<Tenant>[0-9]{4})'}
			}

But apparently since the words are not separated , this doesn't detect it.
Any suggestions ?

2

Hello @WarriorHarb,
go to Kibana - Dev Tools - Grok Debugger

Sample data

AgentID=50119&Tenant=2003&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=000000000

Grok pattern

%{WARRIOR_GROK:debug}

Custom patterns

ALL_EXCEPT_AMP [^\&]*
WARRIOR_GROK ^AgentID=%{ALL_EXCEPT_AMP}&Tenant=%{ALL_EXCEPT_AMP:tenant}

Output

{
  "debug": "AgentID=50119&Tenant=2003",
  "tenant": "2003"
}
3

GROK REGEXP
PUT GROK regexp to a file in directory /etc/logstash/grok-patterns

#vim /etc/logstash/grok-patterns/warrior
ALL_EXCEPT_AMP [^\&]*
WARRIOR_GROK ^AgentID=%{ALL_EXCEPT_AMP}&Tenant=%{ALL_EXCEPT_AMP:tenant}

FILTER GROK PLUGIN
In your logstash pipeline set the filter grok plugin.

  grok {
    patterns_dir => ["/etc/logstash/grok-patterns"]
    break_on_match => false
    match => { "message" => [ "%{WARRIOR_GROK}" ] }
    overwrite => [ "message" ]
    tag_on_failure => ["parsing_failure_on_grok_warrior"]
  }
4

Hello @vasek thanks for your quick response,
The problem is that this parameter isn't always at the same place,

DialedNumber=000000000&ID=843&CountryCode=00&AreaCode=33&Tenant=2002&AgentID=0&GroupID=0&ScriptSource=1&AgentLeg=0

Also in some lines the Tenant doesn't exist at all

5

Have you considered using the kv filter instead of the grok filter? If you are only interested in the tenant field, you could then use the include_keys option to only include that.

6

hi @BennyInc i got many other fields to detect with the same concept

7

You can try this:

ALL_EXCEPT_AMP [^\&]*
WARRIOR_GROK (AgentID=%{ALL_EXCEPT_AMP:agent_id}|Tenant=%{ALL_EXCEPT_AMP:tenant}|GroupID=%{ALL_EXCEPT_AMP:groupd_id}|CountryCode=%{ALL_EXCEPT_AMP:country_code}|AreaCode=%{ALL_EXCEPT_AMP:area_code}|DialedNumber=%{ALL_EXCEPT_AMP:dialed_number}|&)*

It doesn't matter on order and parameters can be empty or not present.

8

grok is not the right tool for this. Use a kv filter

kv { field_split => "&" include_keys => [ "Tenant" ] }

The include_keys option is optional, by default it will extract every key/value pair. There is also an exclude_keys option.

1 Like
9

Hi @Badger,
it worked fine when the tenant is in the middle but when it is in the beginning like this :

`	Tenant=2003&AgentID=50119&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=00000000`

it isn't detected at all.
And in the case Tenant is in the end it detected everything in the rest of log line after the space like in this case :

AgentID=90610&AgentName=0ec3eb9e1159928d21e722996979d4e3&GroupID=19983&AgentLeg=1&DialedNumber=%2b33969780004&CountryCode=33&AreaCode=&Tenant=2002  Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5)

The Tenant value was : 2002 Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5)

that's right 2002 is the tenant ID but the rest is not correct.

10

I would guess towards

kv { field_split => "&" trim_key => "` " }

but if you could provide an actual event and actual filter that does not work and an explanation of what is wrong with it that would help

11

This is i guess what you asked me for:
an example of a log line :

2019-05-25 01:21:03 W3SVC1 127.0.0.1 GET /DialPlan/FT_CheckPlan.aspx DialedNumber=%0000000000&Tenant=2138&ID=582&CountryCode=00&AreaCode=&AgentID=54567&GroupID=16707&ScriptSource=0&AgentLeg=1 Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5)

and another exemple witha different position of tenant :

2019-05-25 01:22:49 W3SVC1 127.0.0.1 GET /DialPlan/FT_CheckPlan.aspx Tenant=2003&AgentID=50119&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=00000000  Mozilla/5.0+(Windows+NT;+Windows+NT+6.3;+en-US)+WindowsPowerShell

and Tenant could be at the end also followed by a space and then the rest of the log line (in this case the tenant value would be fetched but also the rest of the log line)

After applying the grok filter and parsing the whole line into fields, i extract this whole part in a variable messageA :

Tenant=2003&AgentID=50119&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=00000000

then i would like to extract the Tenant value.
I applied as u suggested the KV filter :

kv { field_split => "&" include_keys => [ "Tenant" ] }

it extracted the tenant ID when it was located in the middle, but when it's in the beginning ,the ID isn't detected.

12

You would need to make sure the kv filter uses the correct source field:

kv { field_split => "&" include_keys => [ "Tenant" ] source => "messageA" }

By default it reads from the message field.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.