# Add a field from one grok match to another

**URL:** <https://discuss.elastic.co/t/add-a-field-from-one-grok-match-to-another/288246>\
**Category:** Logstash\
**Created:** [November 2, 2021, 4:25pm UTC](https://discuss.elastic.co/t/add-a-field-from-one-grok-match-to-another/288246 "2021-11-02T16:25:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mwas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mwas/32/77188_2.png) [@mwas](https://discuss.elastic.co/u/mwas)\
**Post date:** [November 2, 2021, 4:25pm UTC](https://discuss.elastic.co/t/add-a-field-from-one-grok-match-to-another/288246/1 "2021-11-02T16:25:46Z")

</div>

I've got log that starts with a jobid on our build cluster live following:  
"Job \<7073381\> is submitted to queue "  
I'd like to extract the job id from that line and add it to all other matched lines that look like this:  
xmelab: \*W,CUSENMP: Use -NAMEMAP\_MIXGEN with name mapped instantiation 'RBR\_SV\_PARAM\_I'

Using following grok for matching the fields:

```auto
grok {
    match => {
        "message" => [ 
        '%{GREEDYDATA} \<%{NUMBER:jobid}\> %{GREEDYDATA} \<%{WORD:queue}\>',
        '%{WORD:process}: %{DATA:log_level},%{DATA:subprocess}: %{GREEDYDATA:logMessage}'
        ]
    }
  }

```

I can't find the way to add value jobid from first match to other matches in the same file.

---

<div class="post-metadata">

**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\
**Post date:** [November 3, 2021, 11:31pm UTC](https://discuss.elastic.co/t/add-a-field-from-one-grok-match-to-another/288246/2 "2021-11-03T23:31:10Z")

</div>

Hi,

From what you tell to us, i can assume that :

- you are using the `file` input plugin. So `Job <7073381> is submitted to queue` and `xmelab: *W,CUSENMP: Use -NAMEMAP_MIXGEN` are two different lines and so differents events.
- You can have other lines who don't respect the two grok pattern you give to us.

To do what you want i think you have to use a ruby filter to save the `job_id` value to use it in other lines.

```auto
grok {
  match => {
    "message" => '%{GREEDYDATA} \<%{NUMBER:jobid}\> %{GREEDYDATA} \<%{WORD:queue}\>'
  }
  # If the match is verified, then add 'source_job_id_line' to the 'tags' field.
  add_tag => ["source_job_id_line"] 
}

grok {
  match => {
    "message" => '%{WORD:process}: %{DATA:log_level},%{DATA:subprocess}: %{GREEDYDATA:logMessage}'
  }
  # If the match is verified, then add 'destination_job_id_line' to the 'tags' field.
  add_tag => ["destination_job_id_line"]
}

# If the current line contains a new job id
if 'source_job_id_line' in [tags] {
  ruby {
    # Initialization of jobId to -1 at logstash startup-time 
    init => '@@jobId = -1'
    # Put the content of the field 'jobid' to the class variable
    code => '
      @@jobId = event.get('jobid');
    '
    remove_tag => ['source_job_id_line']
  }
}

# If the current line need the job id
if 'destination_job_id_line' in [tags] {
  ruby {
    code => '
      # Adding the job id to the end of the line
      event.set('message', event.get('message') + @@jobId.to_s);
      # Adding a job id field
      event.set('jobid', @@jobId.to_s);
    '
    remove_tag => ['destination_job_id_line']
  }
}

```

What it do :

- First i split the grok filter in two to have the possibility to add a different tag depending of the current line read.
- After, depending of the value in the `tags` field, i edit the `@@jobId` variable or i use it.

**I haven't tried the code so maybe it won't work on the first try. Plus, this configuration need to set the number of pipeline worker to 1 to make sure that logstash read the file line by line in the correct order**

Edit: Like Badger explain in the response, we need to use class variable.

Cad.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 3, 2021, 11:38pm UTC](https://discuss.elastic.co/t/add-a-field-from-one-grok-match-to-another/288246/3 "2021-11-03T23:38:14Z")

</div>

> [@Cad](#):
>
> I never try to use a single Instance Variables in two different ruby plugin, but according to the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency) it should work.

No, each filter instance is a different instance, but you can use a variable with class scope rather than instance scope.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 1, 2021, 11:38pm UTC](https://discuss.elastic.co/t/add-a-field-from-one-grok-match-to-another/288246/4 "2021-12-01T23:38:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.