Need help with restructuring the data using filters

Hi Guys,

Is there a way that we can reference field in a particular event in another event?

Regards
Srini

You might be able to do it using an aggregate filter. Or perhaps an instance or class variable in a ruby filter. It depends on the use case.

I have my data in following pattern:

MasterKey1:
MasterKey2:
MasterKey3:
Child1Key1:
Child1Key2:
Child1Key3:
Child2Key1:
Child2Key2:
Child2Key3:
Child3Key1:
Child3Key2:
Child3Key3:
....
....
....

So there is a keyvalue pair in each line which is master data for all subsequent child values.
I have used multiline and pattern using data starting with space to belong to previous line which ensured all the events related to child are in single line and applied grok on top of that.

My issue now is I need to reference the master keys in every child event . Is that possible?
pattern => "^\s"
what => "previous"
negate => false
max_lines => 1000

Sample data:

MonitoringType: QueueStatistics
QueueManagerXXXXXXXXXXX'
IntervalStartDate: '2018-07-25'
IntervalStartTime: '13.57.01'
IntervalEndDate: '2018-07-25'
IntervalEndTime: '13.58.01'
CommandLevel: 900
ObjectCount: 11
QueueStatistics: 0
QueueName: 'SYSTEM.ADMIN.QMGR.EVENT'
CreateDate: '2018-06-08'
CreateTime: '12.53.38'
QueueType: Local
QueueDefinitionType: Predefined
QMinDepth: 0
QMaxDepth: 0
AverageQueueTime: [0, 0]
QueueStatistics: 1
QueueName: 'SYSTEM.CLUSTER.COMMAND.QUEUE'
CreateDate: '2018-06-08'
CreateTime: '12.53.38'
QueueType: Local
QueueDefinitionType: Predefined
QMinDepth: 0
QMaxDepth: 0
AverageQueueTime: [0, 0]
QueueStatistics: 2
QueueName: 'SYSTEM.INTER.QMGR.PUBS'
CreateDate: '2018-06-08'
CreateTime: '12.53.38'
QueueType: Local
QueueDefinitionType: Predefined
QMinDepth: 0
QMaxDepth: 0
AverageQueueTime: [0, 0]
QueueStatistics: 3
QueueName: 'SYSTEM.BROKER.ADMIN.STREAM'
CreateDate: '2018-06-08'
CreateTime: '12.53.38'
QueueType: Local
QueueDefinitionType: Predefined
QMinDepth: 0
QMaxDepth: 0
AverageQueueTime: [0, 0]

Regards
Srinivasa

I assume the QueueManager entry is really something like

QueueManager: 'XXXXXXXXXXX'

If it is not then you should gsub it until it is.

I would read that file using a multiline codec

file {
    path => "/path/to/file.txt"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline { pattern => "^(QueueManager|QueueName)" negate => true what => "previous" auto_flush_interval => 2 }
}

The use a kv filter to split the lines back up.

    kv {
        field_split => "
"
        value_split => ":"
    }

Finally a class variable to add the queue manager name to each queue. This requires that you set

--pipeline.workers 1

and then in your config...

    if [MonitoringType] { drop{} }

    # Stash Queue Manager name 
    if [QueueManager] { ruby { code => '@@qm = event.get("QueueManager"); puts @@qm' } }
    
    # and add it to queues
    if [QueueName]    { ruby { code => 'event.set("QueueManager", @@qm)' } }

Then you can use mutate+add_field to combine dates and times and then a date filter to parse them to timestamps. And possibly a mutate+split to split AverageQueueTime to an array.

If you want to use the IntervalEndDate/IntervalEndTime as the timestamp on the per-queue events you could stash the date in a different class variable.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.