# Create new field from existing field using REGEX

**URL:** <https://discuss.elastic.co/t/create-new-field-from-existing-field-using-regex/204709>\
**Category:** Logstash\
**Created:** [October 22, 2019, 6:26pm UTC](https://discuss.elastic.co/t/create-new-field-from-existing-field-using-regex/204709 "2019-10-22T18:26:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marta\_Zagrajek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marta_zagrajek/32/47442_2.png) [@Marta\_Zagrajek](https://discuss.elastic.co/u/Marta_Zagrajek)\
**Post date:** [October 22, 2019, 6:26pm UTC](https://discuss.elastic.co/t/create-new-field-from-existing-field-using-regex/204709/1 "2019-10-22T18:26:50Z")

</div>

I have grok like that from apache logs:

> grok {  
> match =\> { "message" =\> '%{IP:ip}\s(%{GREEDYDATA:logname})?\s(%{GREEDYDATA:remote\_user})?\s[%{HTTPDATE:timestamp}]\s"%{WORD:metoda}\s%{NOTSPACE:request}\s(HTTP/%{NUMBER:httpversion}")?\s%{NUMBER:response}\s%{GREEDYDATA:bytes}\s("%{GREEDYDATA:referer}")?\s("%{GREEDYDATA:user\_setup}")?\s%{GREEDYDATA:canonical}' }  
> remove\_field =\> ["message"]  
> }
> 
> if [request] == "nagios.plc" {  
> drop {}  
> }

This is working perfectly fine.

I want to extract from request (extracted in match from message field) another two fields using REGEX.

I.E. request: /im/56/d9/17/z25006678F,Barka-hotelowa-na-Wisle.jpg  
I want:  
id: **25006678**  
Format: **F**  
I tried something like that:

> ```
> grok {
> match => { "message" => '%{IP:ip}\s(%{GREEDYDATA:logname})?\s(%{GREEDYDATA:remote_user})?\s\[%{HTTPDATE:timestamp}\]\s\"%{WORD:metoda}\s%{NOTSPACE:request}\s(HTTP/%{NUMBER:httpversion}\")?\s%{NUMBER:response}\s%{GREEDYDATA:bytes}\s(\"%{GREEDYDATA:referer}\")?\s(\"%{GREEDYDATA:user_setup}\")?\s%{GREEDYDATA:canonical}' }
> remove_field => ["message"]
> }
> if [request] == "nagios.plc" {
> drop {}
> }
> grok {
> match => {"request" => "XX: (?<request>([0-9]*))" }
> match => {"request" => "Format: (?<request>(\D+).jpg") }
> 
> ```

Then I tried to move it before removing the message in first grok and replacing first "request" with message as logstash might not know request at that time, but logs still were saying something is wrong with those matches.

Should it be one match instead 3 ?  
Should I use different plugin for logstash?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 22, 2019, 8:07pm UTC](https://discuss.elastic.co/t/create-new-field-from-existing-field-using-regex/204709/2 "2019-10-22T20:07:51Z")

</div>

You have a field that contains /im/56/d9/17/z25006678F,Barka-hotelowa-na-Wisle.jpg  
and you are wondering why neither "XX: (?([0-9]\*))" nor "Format: (?(\D+).jpg" match that. Is that your question?

---

<div class="post-metadata">

**Author:** ![Marta\_Zagrajek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marta_zagrajek/32/47442_2.png) [@Marta\_Zagrajek](https://discuss.elastic.co/u/Marta_Zagrajek)\
**Post date:** [October 24, 2019, 6:25am UTC](https://discuss.elastic.co/t/create-new-field-from-existing-field-using-regex/204709/3 "2019-10-24T06:25:55Z")

</div>

When I am trying that regexes in [https://regex101.com/](https://regex101.com/) they perfectly catching those groups from request.  
I do not know if:

- my conf is proper with so many single matches
- I should use more complex regex to catch everything before as well and everything will be OK ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 24, 2019, 2:19pm UTC](https://discuss.elastic.co/t/create-new-field-from-existing-field-using-regex/204709/4 "2019-10-24T14:19:46Z")

</div>

> [@Marta\_Zagrajek](#):
>
> When I am trying that regexes in [https://regex101.com/](https://regex101.com/) they perfectly catching those groups from request.

No, they are not. In the window on the right it says "Your regular expression does not match the subject string." This is why grok does not create any fields: the pattern does not match. Note the "XX: matches the characters XX: literally (case sensitive)" which is not coloured. The characters "XX: " do not occur in your string so they do not match.

If you remove the "Format: " your second pattern will match "F,Barka-hotelowa-na-Wisle.jpg". I think you want

```
    grok { match => { "request" => "z(?<r>[0-9]+)" } }
    grok { match => { "request" => "(?<s>\D),\D+\.jpg" } }

```

If you just remove the XX: then your `(?<request>([0-9]*))` matches your string 43 different times, and grok would just return the first one. \* means zero or more and  
in "/im/56/d9/17/z25006678F,Barka-hotelowa-na-Wisle.jpg" there are zero digits before the leading /, so it matches there (and grok will match a nil string there), then there are zero digits before the i, so it matches there, zero before the m, so it matches there, and zero before the next / so it matches there again. Then it matches 56, and so on. If you change it to `(?<request>([0-9]+))` then it will match 56. You might be able to use `"(?<r>[0-9]{3,})"`, which means at least 3 digits.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2019, 2:19pm UTC](https://discuss.elastic.co/t/create-new-field-from-existing-field-using-regex/204709/5 "2019-11-21T14:19:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.