Create new field based on msg filed in logstash


(Anil Kumar) #1

Hi All,

Here i want to create a new field Invoice_IID based on msg filed, contains Invoice_IID value in log line msg filed.

 "msg" => "Finished Creating Parent Invoices for Invoice_IID: 80000000-41fb-1638-cd42-ffff08d24480"

My configuration is:

filter {

grok { 
  match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg}" } 
}

if "Invoice_IID" in [msg] {
 mutate {
  add_field => { "Invoice_IID" => "%{msg}" }
}
}
}

This configuration create just a field, but i want the data of Invoice_IID value like 80000000-41fb-1638-cd42-ffff08d24480


Multiple grok matching fileds
(Magnus Bäck) #2

Use another grok filter that matches against the msg field and extracts the invoice id field.


(Anil Kumar) #3
grok { 
  match => { "msg" => "%{GREEDYDATA:text}"} 
}

this is msg field value

Processing 0 promo lines for Invoice_IID 80000000-109d-15f9-f17c-ffff08d24505

How to write the pattern fro above line again in another grok filter


(Anil Kumar) #4

Yes @magnusbaeck. I split the msg field value into below pattern.

msg filed value find above.

grok { 
  match => { "msg" => "%{GREEDYDATA:text}%{UUID:uuid}"} 
}

But how, I assess the which one is InvoiceIID


(Magnus Bäck) #5

Why are you using GREEDYDATA here? That's exactly what's causing the expression to extract any UUID found in any message. Use this instead:

^Processing ${INT} promo lines for Invoice_IID %{UUID:uuid}

(Anil Kumar) #6

Hi @magnusbaeck,

I have two lines like

Processing 0 promo lines for Invoice_IID 80000000-41fb-1638-cd42-ffff08d24480
Processed Inovoice_IID: 80000000-41fb-1638-cd42-ffff08d24480 successfully. 

My Configuration is:

grok { 
  match => { "msg" => "Invoice_IID: %{UUID:InvoiceIID}"} 
}

But here am getting only 2nd line Invoice_IID. Not getting first line Invoice_IID.

I Think the difference is: The colon ( : ) is a punctuation mark

In first line- Invoice_IID 80000000-41fb-1638-cd42-ffff08d24480
In second line- Invoice_IID: 80000000-41fb-1638-cd42-ffff08d24480

I need these two types (if any otherthan these types) are taken into one grok filter with using any OR conditions


(Magnus Bäck) #7

You really should read up on regular expressions. "?" means "zero or one occurrences of the preceding token", i.e. you can use

Invoice_IID:? %{UUID:InvoiceIID}

to make the colon optional.


(Anil Kumar) #8

Hi @magnusbaeck, Can you check once the below Regualr Expression for multiple cases.

multiple cases:

Invoice_IID
InvoiceIID:
Inovoice_IID
invoice iid
invoice_iid

grok { 
  match => { "msg" => [iI]no*voice[_," "][iI][iI][dD]:? %{UUID:InvoiceIID}" } 
}

This is working perfectly in grokDebugger site like http://grokconstructor.appspot.com/do/match#result

But,when am running it's giving a configuration error (chek with --configtest.)

So please provide the correct way of writing RE ??


(Magnus Bäck) #9

Two problems. There's no double quote to start the grok expression, and you have double quotes within your double-quoted string. You should escape those double quotes with a backslash or make the string single-quoted, i.e. use

match => { "msg" => "[iI]no*voice[_,\" \"][iI][iI][dD]:? %{UUID:InvoiceIID}" }

or

match => { "msg" => '[iI]no*voice[_," "][iI][iI][dD]:? %{UUID:InvoiceIID}' }

What is [_, " "] supposed to mean anyway? There's no point in repeating characters within a character class.


(Anil Kumar) #10

Hi @magnusbaeck, Please find the below patterns

cc_digits="1982"  for this am writing  cc_digits="(?<ccdigits>\d{4})"

cc_digits=\"1982\" for this am writing cc_digits=\\"(?<ccdigits>\d{4})\\"

Actually I need both in one regular expression ??

I am trying with below RE

cc_digits=[\\]"(?<ccdigits>\d{4}[)\\]"

This not working properly ????


(Anil Kumar) #11

I got the Solution @magnusbaeck, with using

cc_digits=[\\]*"(?<ccdigits>\d{4})[\\]*"

(system) #12