# Need help with restructuring the data using filters

**URL:** <https://discuss.elastic.co/t/need-help-with-restructuring-the-data-using-filters/142281>\
**Category:** Logstash\
**Created:** [July 31, 2018, 6:06am UTC](https://discuss.elastic.co/t/need-help-with-restructuring-the-data-using-filters/142281 "2018-07-31T06:06:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![whoiscnu](https://avatars.discourse-cdn.com/v4/letter/w/898d66/32.png) [@whoiscnu](https://discuss.elastic.co/u/whoiscnu)\
**Post date:** [July 31, 2018, 6:06am UTC](https://discuss.elastic.co/t/need-help-with-restructuring-the-data-using-filters/142281/1 "2018-07-31T06:06:51Z")

</div>

Hi Guys,

Is there a way that we can reference field in a particular event in another event?

Regards  
Srini

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 2, 2018, 4:35pm UTC](https://discuss.elastic.co/t/need-help-with-restructuring-the-data-using-filters/142281/2 "2018-08-02T16:35:49Z")

</div>

You might be able to do it using an aggregate filter. Or perhaps an instance or class variable in a ruby filter. It depends on the use case.

---

<div class="post-metadata">

**Author:** ![whoiscnu](https://avatars.discourse-cdn.com/v4/letter/w/898d66/32.png) [@whoiscnu](https://discuss.elastic.co/u/whoiscnu)\
**Post date:** [August 3, 2018, 3:52pm UTC](https://discuss.elastic.co/t/need-help-with-restructuring-the-data-using-filters/142281/3 "2018-08-03T15:52:55Z")

</div>

> [@Badger](#):
>
> perhaps an instance or

I have my data in following pattern:

MasterKey1:  
MasterKey2:  
MasterKey3:  
Child1Key1:  
Child1Key2:  
Child1Key3:  
Child2Key1:  
Child2Key2:  
Child2Key3:  
Child3Key1:  
Child3Key2:  
Child3Key3:  
....  
....  
....

So there is a keyvalue pair in each line which is master data for all subsequent child values.  
I have used multiline and pattern using data starting with space to belong to previous line which ensured all the events related to child are in single line and applied grok on top of that.

My issue now is I need to reference the master keys in every child event . Is that possible?  
pattern =\> "^\s"  
what =\> "previous"  
negate =\> false  
max\_lines =\> 1000

Sample data:

MonitoringType: QueueStatistics  
QueueManagerXXXXXXXXXXX'  
IntervalStartDate: '2018-07-25'  
IntervalStartTime: '13.57.01'  
IntervalEndDate: '2018-07-25'  
IntervalEndTime: '13.58.01'  
CommandLevel: 900  
ObjectCount: 11  
QueueStatistics: 0  
QueueName: 'SYSTEM.ADMIN.QMGR.EVENT'  
CreateDate: '2018-06-08'  
CreateTime: '12.53.38'  
QueueType: Local  
QueueDefinitionType: Predefined  
QMinDepth: 0  
QMaxDepth: 0  
AverageQueueTime: [0, 0]  
QueueStatistics: 1  
QueueName: 'SYSTEM.CLUSTER.COMMAND.QUEUE'  
CreateDate: '2018-06-08'  
CreateTime: '12.53.38'  
QueueType: Local  
QueueDefinitionType: Predefined  
QMinDepth: 0  
QMaxDepth: 0  
AverageQueueTime: [0, 0]  
QueueStatistics: 2  
QueueName: 'SYSTEM.INTER.QMGR.PUBS'  
CreateDate: '2018-06-08'  
CreateTime: '12.53.38'  
QueueType: Local  
QueueDefinitionType: Predefined  
QMinDepth: 0  
QMaxDepth: 0  
AverageQueueTime: [0, 0]  
QueueStatistics: 3  
QueueName: 'SYSTEM.BROKER.ADMIN.STREAM'  
CreateDate: '2018-06-08'  
CreateTime: '12.53.38'  
QueueType: Local  
QueueDefinitionType: Predefined  
QMinDepth: 0  
QMaxDepth: 0  
AverageQueueTime: [0, 0]

Regards  
Srinivasa

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 3, 2018, 5:30pm UTC](https://discuss.elastic.co/t/need-help-with-restructuring-the-data-using-filters/142281/4 "2018-08-03T17:30:40Z")

</div>

I assume the QueueManager entry is really something like

```
QueueManager: 'XXXXXXXXXXX'

```

If it is not then you should gsub it until it is.

I would read that file using a multiline codec

```
file {
    path => "/path/to/file.txt"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline { pattern => "^(QueueManager|QueueName)" negate => true what => "previous" auto_flush_interval => 2 }
}

```

The use a kv filter to split the lines back up.

```
    kv {
        field_split => "
"
        value_split => ":"
    }

```

Finally a class variable to add the queue manager name to each queue. This requires that you set

```
--pipeline.workers 1

```

and then in your config...

```
    if [MonitoringType] { drop{} }

    # Stash Queue Manager name 
    if [QueueManager] { ruby { code => '@@qm = event.get("QueueManager"); puts @@qm' } }
    
    # and add it to queues
    if [QueueName] { ruby { code => 'event.set("QueueManager", @@qm)' } }

```

Then you can use mutate+add\_field to combine dates and times and then a date filter to parse them to timestamps. And possibly a mutate+split to split AverageQueueTime to an array.

If you want to use the IntervalEndDate/IntervalEndTime as the timestamp on the per-queue events you could stash the date in a different class variable.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 31, 2018, 5:30pm UTC](https://discuss.elastic.co/t/need-help-with-restructuring-the-data-using-filters/142281/5 "2018-08-31T17:30:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.