# Extracting a new field from the message field **URL:** \ **Category:** Logstash\ **Created:** [March 6, 2020, 6:06am UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407 "2020-03-06T06:06:09Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Hari\_Krishna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_krishna/32/62678_2.png) [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Post date:** [March 6, 2020, 6:06am UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/1 "2020-03-06T06:06:10Z")
2020-03-06 09:36:52.801907464 re0:ndp:25786 lltp\_debug message = "NDP-DBG:NC\_FSM\_HANDLER:1634:: state 7, event 4, NexthopId 44001" This is my log. I need to extract state and NexthopId from the message field match =\> { "message" =\>"%{TIMESTAMP\_ISO8601:timestamp} %{WORD:node}:%{WORD:program}:%{INT:pid} %{WORD:tracetype}._%{mssg:Message}._"} custom pattern mssg ((Msg|message|Message|message1|message2) [=] ["]%{DATA}+["]) ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 6, 2020, 2:00pm UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/2 "2020-03-06T14:00:06Z")
Hi there, do you need to extract only the `NexthopId` or all the other fields you put in your grok, too? ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 6, 2020, 2:19pm UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/3 "2020-03-06T14:19:04Z")
> [@Hari\_Krishna](#): > > %{TIMESTAMP\_ISO8601:timestamp} %{WORD:node}:%{WORD:program}:%{INT:pid} %{WORD:tracetype}. _%{mssg:Message}._ "} Anyway, what about this: ``` filter { grok { break_on_match => false pattern_definitions => { "mssg" => "(Msg|message|Message|message1|message2) [=]" } match => { "message" => ["%{TIMESTAMP_ISO8601:timestamp} %{WORD:node}:%{WORD:program}:%{INT:pid} %{WORD:tracetype} %{mssg} \"%{GREEDYDATA:Message}\""] "Message" => ["NexthopId %{WORD:next_hop_id}"] } } } ``` Obviously, if you're not sure you can to extract the `Message` field from every event, you can make two grok filters and put the second one (the one on `Message`) in a condition like `if [Message] { ...grok filter...}`. Also, just check if there's any event with a different pattern which breaks the first grok. That is up to you. ---
**Author:** ![Hari\_Krishna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_krishna/32/62678_2.png) [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Post date:** [March 7, 2020, 8:00am UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/4 "2020-03-07T08:00:23Z")
Hi Fabio, I tried your filter. But when i run logstash, it uses the default mapping template. There is no error in the conf file. Can you help me out And I am new to logstash 🙂 ---
**Author:** ![Hari\_Krishna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_krishna/32/62678_2.png) [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Post date:** [March 7, 2020, 8:01am UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/5 "2020-03-07T08:01:23Z")
This is my config file input { file { path =\> ["/home/hari/ndppro/ndp2a.log"] start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { grok { patterns\_dir =\> ["/etc/logstash/pattern"] match =\> { "message" =\> ["%{TIMESTAMP\_ISO8601:timestamp} %{WORD:node}:%{WORD:program}:%{INT:pid} %{WORD:tracetype} %{mssg} "%{GREEDYDATA:Message}""] "Message" =\> ["NexthopId %{WORD:next\_hop\_id}"] } } mutate { remove\_field =\> ["message"] } } output { elasticsearch{ hosts =\> ["localhost:9200"] } } ~ ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 7, 2020, 9:23am UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/6 "2020-03-07T09:23:51Z")
So, first of all when posting some code (or anything which is not plain writing) please format it, or it'll be impossible to read for us. So use any editor (VSCode, Atom, Sublime or whatever) to properly indent your code, paste it here **properly indented** , highlight it and click on the **`Preformatted tool`** ( ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fc138e622748bafe24dc250e7af179640871a7a1.png) ). Speaking of your question, I don't really get what you mean by > it uses the default mapping template Plus, if I insert something in my pipeline (like the `break_on_match => false` in the grok, why did you remove it? Finally, can you post here some outputs of the following pipeline (output will be in your standard output, so your terminal): ``` input { file { path => ["/home/hari/ndppro/ndp2a.log"] start_position => "beginning" sincedb_path => "/dev/null" } } filter { grok { break_on_match => false pattern_definitions => { "mssg" => "(Msg|message|Message|message1|message2) [=]" } match => { "message" => ["%{TIMESTAMP_ISO8601:timestamp} %{WORD:node}:%{WORD:program}:%{INT:pid} %{WORD:tracetype} %{mssg} "%{GREEDYDATA:Message}""] "Message" => ["NexthopId %{WORD:next_hop_id}"] } } mutate { remove_field => ["message"] } } output { stdout{} } ``` ---
**Author:** ![Hari\_Krishna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_krishna/32/62678_2.png) [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Post date:** [March 7, 2020, 5:18pm UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/7 "2020-03-07T17:18:41Z")
Hi Fabio, Thanks. I made some mistake. It works fine now. 😃 Now i have a scenario where i must add a field to display the state name for the corresponding state number. Please find my log below 2020-03-06 09:36:50.775744749 re0:ndp:25786 lltp\_debug message = "NDP-DBG:NC\_FSM\_HANDLER:1634:: state 4, event 0, NexthopId 44000" 2020-03-06 09:36:51.548239404 re0:ndp:25786 lltp\_debug message = "NDP-DBG:NC\_FSM\_HANDLER:1634:: state 5, event 3, NexthopId 44001" 2020-03-06 09:36:52.778379389 re0:ndp:25786 lltp\_debug message = "NDP-DBG:NC\_FSM\_HANDLER:1634:: state 5, event 0, NexthopId 44001" **States** 0 - No state 1 - Unreachable 2 - Incomplete 3 - Reachable 4 - Stale 5 - Delay 6 - Probe Can you help me on how to write the conditions? ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 7, 2020, 6:28pm UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/8 "2020-03-07T18:28:55Z")
Glad it works properly, I do not have access to my laptop right now but I can tell you the steps to follow: - extract the state from the Message field using a grok again with a `break_on_match: false` - use a translate filter or a ruby filter. The translate one is kinda trivial to use (it's basically a dictionary) and you can see an example on the logstash documentation. ---
**Author:** ![Hari\_Krishna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_krishna/32/62678_2.png) [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Post date:** [March 9, 2020, 3:51am UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/9 "2020-03-09T03:51:33Z")
Thanks Fabio. I'll have a look into it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 6, 2020, 3:51am UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407/10 "2020-04-06T03:51:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.