# Create new field based on msg filed in logstash **URL:** \ **Category:** Logstash\ **Created:** [July 14, 2015, 2:29pm UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527 "2015-07-14T14:29:41Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![abathula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abathula/32/3595_2.png) [@abathula](https://discuss.elastic.co/u/abathula)\ **Post date:** [July 14, 2015, 2:29pm UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/1 "2015-07-14T14:29:41Z")
Hi All, Here i want to create a new field **Invoice\_IID** based on **msg** filed, contains **Invoice\_IID** value in log line **msg** filed. ``` "msg" => "Finished Creating Parent Invoices for Invoice_IID: 80000000-41fb-1638-cd42-ffff08d24480" ``` **My configuration is:** ``` filter { grok { match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg}" } } if "Invoice_IID" in [msg] { mutate { add_field => { "Invoice_IID" => "%{msg}" } } } } ``` This configuration create just a field, but i want the data of Invoice\_IID value like 80000000-41fb-1638-cd42-ffff08d24480 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 14, 2015, 2:45pm UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/2 "2015-07-14T14:45:59Z")
Use another grok filter that matches against the `msg` field and extracts the invoice id field. ---
**Author:** ![abathula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abathula/32/3595_2.png) [@abathula](https://discuss.elastic.co/u/abathula)\ **Post date:** [July 14, 2015, 2:48pm UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/3 "2015-07-14T14:48:56Z")
``` grok { match => { "msg" => "%{GREEDYDATA:text}"} } ``` this is **msg** field value ``` Processing 0 promo lines for Invoice_IID 80000000-109d-15f9-f17c-ffff08d24505 ``` How to write the pattern fro above line again in another grok filter ---
**Author:** ![abathula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abathula/32/3595_2.png) [@abathula](https://discuss.elastic.co/u/abathula)\ **Post date:** [July 15, 2015, 5:04am UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/4 "2015-07-15T05:04:50Z")
Yes @magnusbaeck. I split the **msg** field value into below pattern. **msg** filed value find above. ``` grok { match => { "msg" => "%{GREEDYDATA:text}%{UUID:uuid}"} } ``` But how, I assess the which one is InvoiceIID ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 15, 2015, 5:41am UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/5 "2015-07-15T05:41:22Z")
Why are you using GREEDYDATA here? That's exactly what's causing the expression to extract any UUID found in any message. Use this instead: ``` ^Processing ${INT} promo lines for Invoice_IID %{UUID:uuid} ``` ---
**Author:** ![abathula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abathula/32/3595_2.png) [@abathula](https://discuss.elastic.co/u/abathula)\ **Post date:** [July 15, 2015, 7:45am UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/6 "2015-07-15T07:45:44Z")
Hi @magnusbaeck, I have two lines like ``` Processing 0 promo lines for Invoice_IID 80000000-41fb-1638-cd42-ffff08d24480 Processed Inovoice_IID: 80000000-41fb-1638-cd42-ffff08d24480 successfully. ``` **My Configuration is:** ``` grok { match => { "msg" => "Invoice_IID: %{UUID:InvoiceIID}"} } ``` But here am getting only 2nd line Invoice\_IID. Not getting first line Invoice\_IID. I Think the difference is: **The colon ( : ) is a punctuation mark** **In first line-** Invoice\_IID 80000000-41fb-1638-cd42-ffff08d24480 **In second line-** Invoice\_IID: 80000000-41fb-1638-cd42-ffff08d24480 I need these two types ( **if any otherthan these types** ) are taken into one grok filter with using any **OR** conditions ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 15, 2015, 7:51am UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/7 "2015-07-15T07:51:49Z")
You really should read up on regular expressions. "?" means "zero or one occurrences of the preceding token", i.e. you can use ``` Invoice_IID:? %{UUID:InvoiceIID} ``` to make the colon optional. ---
**Author:** ![abathula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abathula/32/3595_2.png) [@abathula](https://discuss.elastic.co/u/abathula)\ **Post date:** [July 15, 2015, 10:21am UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/8 "2015-07-15T10:21:09Z")
Hi @magnusbaeck, Can you check once the below Regualr Expression for multiple cases. **multiple cases:** Invoice\_IID InvoiceIID: Inovoice\_IID invoice iid invoice\_iid ``` grok { match => { "msg" => [iI]no*voice[_," "][iI][iI][dD]:? %{UUID:InvoiceIID}" } } ``` This is working perfectly in **grokDebugger** site like [http://grokconstructor.appspot.com/do/match#result](http://grokconstructor.appspot.com/do/match#result) But,when am running it's giving a configuration error (chek with **--configtest**.) So please provide the correct way of writing **RE**?? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 15, 2015, 10:51am UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/9 "2015-07-15T10:51:54Z")
Two problems. There's no double quote to start the grok expression, and you have double quotes within your double-quoted string. You should escape those double quotes with a backslash or make the string single-quoted, i.e. use ``` match => { "msg" => "[iI]no*voice[_,\" \"][iI][iI][dD]:? %{UUID:InvoiceIID}" } ``` or ``` match => { "msg" => '[iI]no*voice[_," "][iI][iI][dD]:? %{UUID:InvoiceIID}' } ``` What is `[_, " "]` supposed to mean anyway? There's no point in repeating characters within a character class. ---
**Author:** ![abathula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abathula/32/3595_2.png) [@abathula](https://discuss.elastic.co/u/abathula)\ **Post date:** [July 15, 2015, 1:23pm UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/10 "2015-07-15T13:23:04Z")
Hi @magnusbaeck, Please find the below patterns ``` cc_digits="1982" for this am writing cc_digits="(?\d{4})" cc_digits=\"1982\" for this am writing cc_digits=\\"(?\d{4})\\" ``` Actually I need both in one regular expression ?? I am trying with below **RE** ``` cc_digits=[\\]"(?\d{4}[)\\]" ``` This not working properly ???? ---
**Author:** ![abathula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abathula/32/3595_2.png) [@abathula](https://discuss.elastic.co/u/abathula)\ **Post date:** [July 15, 2015, 1:42pm UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/11 "2015-07-15T13:42:51Z")
I got the Solution @magnusbaeck, with using ``` cc_digits=[\\]*"(?\d{4})[\\]*" ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:34am UTC](https://discuss.elastic.co/t/create-new-field-based-on-msg-filed-in-logstash/25527/12 "2017-07-06T05:34:37Z")