# Can't remove\_field - help please **URL:** \ **Category:** Logstash\ **Created:** [February 26, 2016, 9:21pm UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894 "2016-02-26T21:21:22Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Marjana](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Marjana](https://discuss.elastic.co/u/Marjana)\ **Post date:** [February 26, 2016, 9:21pm UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/1 "2016-02-26T21:21:22Z")
I am very new with Logstash, trying to create my first config. I am using 1.5.6 version. As I am testing, I wanted to try to use db2 log file as input. Got the multiline thing working fine. Example of my event: "2016-02-26-14.51.45.577696-360 I959779E406 LEVEL: Warning\nPID : 24767 TID : 140656464664352 PROC : db2star2\nINSTANCE: db2inst1 NODE : 000\nHOSTNAME: [opsmgmt01.wdc01.unica.com](http://opsmgmt01.wdc01.unica.com)\nFUNCTION: DB2 UDB, base sys utilities, sqleReleaseStStLockFile, probe:14016\nMESSAGE : Released lock on the file:\nDATA #1 : String, 39 bytes\n/home/db2inst1/sqllib/ctrl/db2stst.0000" I want to get rid of everything after LEVEL:Warning. This is my filter: filter { grok { match =\> { message =\> "%{YEAR}[-]%{MONTHNUM}[-]%{MONTHDAY}[-]%{HOUR}[.]%{MINUTE}[.]%{SECOND}[-]%{INT}%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel}%{GREEDYDATA:therest}" } } mutate { remove\_field =\> ["therest"] } } I used grok debugger and it displays "therest" correctly. My output: output { if [loglevel] not in ["Info","Event"] { stdout { codec =\> rubydebug } email { to =\> "mivkovic@xyz.com" subject =\> "check db2 instance log" from =\> "root@%{host}" body =\> "Here is the event: %{message}" } } } On my stdout, I see message as the whole thing, including "therest" field. Same thing for my email message body. How does my new message becomes everything but "therest" field? Many thanks! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 26, 2016, 10:29pm UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/2 "2016-02-26T22:29:25Z")
Does your pattern actually match 100%? ---
**Author:** ![mick66](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@mick66](https://discuss.elastic.co/u/mick66)\ **Post date:** [February 27, 2016, 8:42am UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/3 "2016-02-27T08:42:11Z")
I think you may be misunderstanding exactly what the grok filter does. It creates fields from the %{message} based on the pattern provided, but does not alter the original %{message} which will always be what has been received from the input filter, in this case your log entry. Your code groks the message to create fields, but your output filter is sending you the original %{message} which will always contain %{therest}. What you need to do is create a new message from the grok fields and send this instead. Try something like this: ``` match => { message => "(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY}-%{HOUR}.%{MINUTE}.%{SECOND}[-]%{INT})%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel}%{GREEDYDATA:therest}" } mutate { add_field => { "new_message" => "%{runtime}-%{loglevel}"} } output { if [loglevel] not in ["Info","Event"] { stdout { codec => rubydebug } email { to => "mivkovic@xyz.com" subject => "check db2 instance log" from => "root@%{host}" body => "Here is the event: %{new_message}" } } } ``` ---
**Author:** ![Marjana](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Marjana](https://discuss.elastic.co/u/Marjana)\ **Post date:** [February 29, 2016, 4:38pm UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/4 "2016-02-29T16:38:05Z")
Yes it does. Checked it in grok debugger. ---
**Author:** ![Marjana](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Marjana](https://discuss.elastic.co/u/Marjana)\ **Post date:** [February 29, 2016, 4:47pm UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/5 "2016-02-29T16:47:44Z")
Thanks, I was suspecting my "message" always remains the same, in output. What still doesn't make sense to me is that, when I remove\_field in grok, shouldn't stdout { codec =\> rubydebug } not display it? I understand remove\_field like this: takes input and removes the field so it is not shown in output. If not, what does it remove it from then? About your (?\>runtime:...), that doesn't work. I assume you wanted to make it so my whole date/time thing is returned as a runtime field. I have tried to play with various combinations there but none worked. I would think this should work (simplified with just year and month): match =\> { message =\> "%{(%{YEAR}[-]%{MONTHNUM}):runtime} but it doesn't. I followed this: %{SYNTAX:SEMANTIC} so my semantic is runtime. And my syntax is in (), combination of several YEAR and MONTH. Many thanks! ---
**Author:** ![mick66](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@mick66](https://discuss.elastic.co/u/mick66)\ **Post date:** [February 29, 2016, 5:31pm UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/6 "2016-02-29T17:31:07Z")
Sorry I had a typo on the pattern, which I have corrected in the original post above, so you should now get a field called runtime with a value of: `2016-02-26-14.51.45.577696-360` Could you please post an example of the exact output you are getting from the ruby debug. ---
**Author:** ![Marjana](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Marjana](https://discuss.elastic.co/u/Marjana)\ **Post date:** [February 29, 2016, 7:26pm UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/7 "2016-02-29T19:26:15Z")
> [@mick66](#): > > ample of the exact output you are getting from the r Thanks, it works now! This must be the Oniguruma syntax "(?\the pattern here)", right? I had tried that too, but didn't realize I need to have \<\>. 🙂 This is my filter: filter { grok { match =\> { message =\> "(?%{YEAR}[-]%{MONTHNUM}[-]%{MONTHDAY}[-]%{HOUR}[.]%{MINUTE}[.]%{SECOND})[-]%{INT}%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel}%{GREEDYDATA:therest}" } } mutate { remove\_field =\> ["therest"] } mutate { add\_field =\> {"new\_message" =\> "%{runtime}-%{loglevel}"} } } And this is my output: Logstash startup completed { "@timestamp" =\> "2016-02-29T18:44:29.498Z", "message" =\> "2016-02-29-12.44.28.741765-360 E988348E380 LEVEL: Warning (OS)\nPID : 17367 TID : 140140202448672 PROC : db2star2\nINSTANCE: db2inst1 NODE : 000\nHOSTNAME: [opsmgmt01.wdc01.tealeaf.ibmcloud.com](http://opsmgmt01.wdc01.tealeaf.ibmcloud.com)\nFUNCTION: DB2 UDB, SQO Memory Management, sqloMemCreateSingleSegment, probe:100\nCALLED : OS, -, shmget OSERR: EEXIST (17)", "@version" =\> "1", "tags" =\> [ [0] "multiline" ], "host" =\> "[opsmgmt01.xyz.com](http://opsmgmt01.xyz.com)", "path" =\> "/home/db2inst1/sqllib/db2dump/db2diag.log", "type" =\> "db2-log", "runtime" =\> "2016-02-29-12.44.28.741765", "loglevel" =\> "Warning", "new\_message" =\> "2016-02-29-12.44.28.741765-Warning" } I see that all fields are actually returned properly. How would I use remove\_field here, instead of add\_field? So if I wanted to output everything but 1 field, I would think remove\_field would be perfect for it, instead of creating new field to include all but 1 I don't want. Many thanks! Marjana ---
**Author:** ![mick66](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@mick66](https://discuss.elastic.co/u/mick66)\ **Post date:** [March 1, 2016, 10:11am UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/8 "2016-03-01T10:11:19Z")
I am getting a bit confused as to exactly what your requirement is. Your original post had a requirement to generate an email with part of the message as the body of the email. If this is all you need, then what you have should deliver on that requirement. So if you just need to strip off the end of the message, then you could change the example I posted slightly and complete the process using just the grok filter, but this will only create one field: **Example 1:** ``` filter { grok { match => { message => "(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY}-%{HOUR}.%{MINUTE}.%{SECOND}[-]%{INT}%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel})"} } mutate { remove_field => ["message", "loglevel"] } } ``` This will deliver the following output: ``` { "@version" => "1", "@timestamp" => "2016-03-01T09:48:41.659Z", "host" => "server1", "new_message" => "2016-02-26-14.51.45.577696-360 I959779E406 LEVEL: Warning" } ``` This `new_message` field can then be used in the body of your email. If your requirement is to break the message up into individual fields then you already have what you need: **Example 2:** ``` filter { grok { match => { message => "(?%{YEAR}[-]%{MONTHNUM}[-]%{MONTHDAY}[-]%{HOUR}[.]%{MINUTE}[.]%{SECOND})[-]%{INT}%{SPACE}[0-9a-zA-Z]{10,}%{SPACE}+%{WORD}[:]%{SPACE}%{WORD:loglevel}%{GREEDYDATA:therest}" } } mutate { remove_field => ["message"]} } ``` Delivers the following output: ``` { "@version" => "1", "@timestamp" => "2016-03-01T09:54:36.491Z", "host" => "server1", "runtime" => "2016-02-26-14.51.45.577696", "loglevel" => "Warning", "therest" => "\\nPID : 24767 TID : 140656464664352 PROC : db2star2\\nINSTANCE: db2inst1 NODE : 000\\nHOSTNAME: opsmgmt01.wdc01.unica.com\\nFUNCTION: DB2 UDB, base sys utilities, sqleReleaseStStLockFile, probe:14016\\nMESSAGE : Released lock on the file:\\nDATA #1 : String, 39 bytes\\n/home/db2inst1/sqllib/ctrl/db2stst.0000" } ``` So now you can do what you like with these fields and if you need to break up "therest" then just work on your grok filter to split this up further. ---
**Author:** ![Marjana](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@Marjana](https://discuss.elastic.co/u/Marjana)\ **Post date:** [March 1, 2016, 3:29pm UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/9 "2016-03-01T15:29:57Z")
Thanks, this helps a lot!!All set now. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:09am UTC](https://discuss.elastic.co/t/cant-remove-field-help-please/42894/10 "2017-07-06T05:09:06Z")