Create new field based on msg filed in logstash

Hi All,

Here i want to create a new field Invoice_IID based on msg filed, contains Invoice_IID value in log line msg filed.

 "msg" => "Finished Creating Parent Invoices for Invoice_IID: 80000000-41fb-1638-cd42-ffff08d24480"

My configuration is:

filter {

grok { 
  match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg}" } 

if "Invoice_IID" in [msg] {
 mutate {
  add_field => { "Invoice_IID" => "%{msg}" }

This configuration create just a field, but i want the data of Invoice_IID value like 80000000-41fb-1638-cd42-ffff08d24480

Use another grok filter that matches against the msg field and extracts the invoice id field.

grok { 
  match => { "msg" => "%{GREEDYDATA:text}"} 

this is msg field value

Processing 0 promo lines for Invoice_IID 80000000-109d-15f9-f17c-ffff08d24505

How to write the pattern fro above line again in another grok filter

Yes @magnusbaeck. I split the msg field value into below pattern.

msg filed value find above.

grok { 
  match => { "msg" => "%{GREEDYDATA:text}%{UUID:uuid}"} 

But how, I assess the which one is InvoiceIID

Why are you using GREEDYDATA here? That's exactly what's causing the expression to extract any UUID found in any message. Use this instead:

^Processing ${INT} promo lines for Invoice_IID %{UUID:uuid}

Hi @magnusbaeck,

I have two lines like

Processing 0 promo lines for Invoice_IID 80000000-41fb-1638-cd42-ffff08d24480
Processed Inovoice_IID: 80000000-41fb-1638-cd42-ffff08d24480 successfully. 

My Configuration is:

grok { 
  match => { "msg" => "Invoice_IID: %{UUID:InvoiceIID}"} 

But here am getting only 2nd line Invoice_IID. Not getting first line Invoice_IID.

I Think the difference is: The colon ( : ) is a punctuation mark

In first line- Invoice_IID 80000000-41fb-1638-cd42-ffff08d24480
In second line- Invoice_IID: 80000000-41fb-1638-cd42-ffff08d24480

I need these two types (if any otherthan these types) are taken into one grok filter with using any OR conditions

You really should read up on regular expressions. "?" means "zero or one occurrences of the preceding token", i.e. you can use

Invoice_IID:? %{UUID:InvoiceIID}

to make the colon optional.

Hi @magnusbaeck, Can you check once the below Regualr Expression for multiple cases.

multiple cases:

invoice iid

grok { 
  match => { "msg" => [iI]no*voice[_," "][iI][iI][dD]:? %{UUID:InvoiceIID}" } 

This is working perfectly in grokDebugger site like

But,when am running it's giving a configuration error (chek with --configtest.)

So please provide the correct way of writing RE ??

Two problems. There's no double quote to start the grok expression, and you have double quotes within your double-quoted string. You should escape those double quotes with a backslash or make the string single-quoted, i.e. use

match => { "msg" => "[iI]no*voice[_,\" \"][iI][iI][dD]:? %{UUID:InvoiceIID}" }


match => { "msg" => '[iI]no*voice[_," "][iI][iI][dD]:? %{UUID:InvoiceIID}' }

What is [_, " "] supposed to mean anyway? There's no point in repeating characters within a character class.

Hi @magnusbaeck, Please find the below patterns

cc_digits="1982"  for this am writing  cc_digits="(?<ccdigits>\d{4})"

cc_digits=\"1982\" for this am writing cc_digits=\\"(?<ccdigits>\d{4})\\"

Actually I need both in one regular expression ??

I am trying with below RE


This not working properly ????

I got the Solution @magnusbaeck, with using