# Create temp metadata field using regex in message field and parse json

**URL:** <https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594>\
**Category:** Logstash\
**Created:** [August 11, 2020, 4:16pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594 "2020-08-11T16:16:18Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\
**Post date:** [August 11, 2020, 4:16pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594/1 "2020-08-11T16:16:19Z")

</div>

I need to extract the JSON object from the message field

```
filter {
    grok { 
        match => { "message" => "(?<[@metadata][tempjson]>{.+})" } 
    }
    json {
        source => "[@metadata][tempjson]"
    }
}

```

Above filter works in 7.8 but in production, we are on 6.4.1 and above filter throws the following error:

> Aug 11 10:49:25 logstash[24400]: [2020-08-11T10:49:25,354][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#\<LogStash::FilterDelegator:0x7e242ff9 @metric\_events\_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric\_events\_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric\_events\_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration\_in\_millis value:0, @id="0b558f1ec526e9beddd9873771d6477a6f55d61976c1fb340737ebd85e3e5120", @klass=LogStash::Filters::Grok, @metric\_events=#LogStash::Instrument::NamespacedMetric:0x32a505ed, @filter=\<LogStash::Filters::Grok match=\>{"message"=\>"(?\<[@metadata][tempjson]\>{.+})"}, id=\>"0b558f1ec526e9beddd9873771d6477a6f55d61976c1fb340737ebd85e3e5120", enable\_metric=\>true, periodic\_flush=\>false, patterns\_files\_glob=\>"\*", break\_on\_match=\>true, named\_captures\_only=\>true, keep\_empty\_captures=\>false, tag\_on\_failure=\>["\_grokparsefailure"], timeout\_millis=\>30000, tag\_on\_timeout=\>"\_groktimeout"\>\>", :error=\>"invalid char in group name \<[@metadata][tempjson]\>: /(?\<[@metadata][tempjson]\>{.+})/m", :thread=\>"#\<Thread:0x2adaf67d run\>"}
> 
> Aug 11 10:49:25 logstash[24400]: [2020-08-11T10:49:25,607][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<RegexpError: invalid char in group name \<[@metadata][tempjson]\>: /(?\<[@metadata][tempjson]\>{.+})/m\>, :backtrace=\>["org/jruby/RubyRegexp.java:928:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:127:in `compile'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:281:in `block in register'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:275:in `block in register'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:270:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:242:in `register\_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:595:in `maybe\_setup\_out\_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:263:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:200:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:160:in `block in start'"], :thread=\>"#\<Thread:0x2adaf67d run\>"}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 11, 2020, 5:00pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594/2 "2020-08-11T17:00:09Z")

</div>

Does it work in 6.4.1 if you just put the field at the top level instead of trying to put it under [@metadata], that is

```
"(?<tempjson>{.+})"
```

---

<div class="post-metadata">

**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\
**Post date:** [August 11, 2020, 5:03pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594/3 "2020-08-11T17:03:43Z")

</div>

Yup thats what I'm doing right now, and after parsing I'm removing that temp field, but I'm hoping to avoid step where I need to remove this temp field. Metadata is exactly what I need but its not working in production.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 11, 2020, 5:08pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594/4 "2020-08-11T17:08:34Z")

</div>

If you add 'remove\_field =\> ["tempjson"]' to the json filter it will be left on the event only if it is not valid JSON. That might be useful.

---

<div class="post-metadata">

**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\
**Post date:** [August 11, 2020, 5:16pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594/5 "2020-08-11T17:16:40Z")

</div>

I'm adding `remove_field => ["tempjson"]` after parsing like this:

```
      grok { 
        match => { "message" => "(?<tempjson>{.+})" } 
      }

      json {
        source => "tempjson"
      }

      mutate {
        remove_field => ["tempjson"]
      }
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 11, 2020, 6:08pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594/6 "2020-08-11T18:08:03Z")

</div>

That will work, but it means the temporary field is unconditionally removed, and if there is ever JSON that fails to parse you will not be able to see what it is. That is why I suggested moving the remove\_field =\> ["tempjson"] to the json filter.

---

<div class="post-metadata">

**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\
**Post date:** [August 11, 2020, 6:15pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594/7 "2020-08-11T18:15:12Z")

</div>

Sure makes sense, I will do that thank you 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 8, 2020, 6:15pm UTC](https://discuss.elastic.co/t/create-temp-metadata-field-using-regex-in-message-field-and-parse-json/244594/8 "2020-09-08T18:15:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.