Create new field based on msg filed in logstash

Elastic Stack Logstash
1

Hi All,

Here i want to create a new field Invoice_IID based on msg filed, contains Invoice_IID value in log line msg filed.

 "msg" => "Finished Creating Parent Invoices for Invoice_IID: 80000000-41fb-1638-cd42-ffff08d24480"

My configuration is:

filter {

grok { 
  match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg}" } 
}

if "Invoice_IID" in [msg] {
 mutate {
  add_field => { "Invoice_IID" => "%{msg}" }
}
}
}

This configuration create just a field, but i want the data of Invoice_IID value like 80000000-41fb-1638-cd42-ffff08d24480

2

Use another grok filter that matches against the msg field and extracts the invoice id field.

3 
grok { 
  match => { "msg" => "%{GREEDYDATA:text}"} 
}

this is msg field value

Processing 0 promo lines for Invoice_IID 80000000-109d-15f9-f17c-ffff08d24505

How to write the pattern fro above line again in another grok filter

4

Yes @magnusbaeck. I split the msg field value into below pattern.

msg filed value find above.

grok { 
  match => { "msg" => "%{GREEDYDATA:text}%{UUID:uuid}"} 
}

But how, I assess the which one is InvoiceIID

5

Why are you using GREEDYDATA here? That's exactly what's causing the expression to extract any UUID found in any message. Use this instead:

^Processing ${INT} promo lines for Invoice_IID %{UUID:uuid}
6

Hi @magnusbaeck,

I have two lines like

Processing 0 promo lines for Invoice_IID 80000000-41fb-1638-cd42-ffff08d24480
Processed Inovoice_IID: 80000000-41fb-1638-cd42-ffff08d24480 successfully.

My Configuration is:

grok { 
  match => { "msg" => "Invoice_IID: %{UUID:InvoiceIID}"} 
}

But here am getting only 2nd line Invoice_IID. Not getting first line Invoice_IID.

I Think the difference is: The colon ( : ) is a punctuation mark

In first line- Invoice_IID 80000000-41fb-1638-cd42-ffff08d24480
In second line- Invoice_IID: 80000000-41fb-1638-cd42-ffff08d24480

I need these two types (if any otherthan these types) are taken into one grok filter with using any OR conditions

7

You really should read up on regular expressions. "?" means "zero or one occurrences of the preceding token", i.e. you can use

Invoice_IID:? %{UUID:InvoiceIID}

to make the colon optional.

8

Hi @magnusbaeck, Can you check once the below Regualr Expression for multiple cases.

multiple cases:

Invoice_IID
InvoiceIID:
Inovoice_IID
invoice iid
invoice_iid

grok { 
  match => { "msg" => [iI]no*voice[_," "][iI][iI][dD]:? %{UUID:InvoiceIID}" } 
}

This is working perfectly in grokDebugger site like http://grokconstructor.appspot.com/do/match#result

But,when am running it's giving a configuration error (chek with --configtest.)

So please provide the correct way of writing RE ??

9

Two problems. There's no double quote to start the grok expression, and you have double quotes within your double-quoted string. You should escape those double quotes with a backslash or make the string single-quoted, i.e. use

match => { "msg" => "[iI]no*voice[_,\" \"][iI][iI][dD]:? %{UUID:InvoiceIID}" }

or

match => { "msg" => '[iI]no*voice[_," "][iI][iI][dD]:? %{UUID:InvoiceIID}' }

What is [_, " "] supposed to mean anyway? There's no point in repeating characters within a character class.

10

Hi @magnusbaeck, Please find the below patterns

cc_digits="1982"  for this am writing  cc_digits="(?<ccdigits>\d{4})"

cc_digits=\"1982\" for this am writing cc_digits=\\"(?<ccdigits>\d{4})\\"

Actually I need both in one regular expression ??

I am trying with below RE

cc_digits=[\\]"(?<ccdigits>\d{4}[)\\]"

This not working properly ????

11

I got the Solution @magnusbaeck, with using

cc_digits=[\\]*"(?<ccdigits>\d{4})[\\]*"