Elapsed plugin and multiple files/matches


(CPS) #1

Hello

In logstash config I would like to use elapsed filter to match particular ID which can exist in 3 separate FILES:

filter {
        if ([path] =~ "FILEA")
        {
		grok{
		      match => ["message","RECEIVED SOME MESSAGE ID: %{NUMBERLC:tradeid}"]
			add_tag => [ "firstTaskStarted" ]
		}

		grok{                  
			match => ["message","PUBLISHED MESSAGE ID: %{NUMBERLC:tradeid}"]
			add_tag => [ "taskTerminated" ]
		}
		elapsed
			{
                   start_tag => "firstTaskStarted"
                  end_tag => "taskTerminated"
                 unique_id_field => "tradeid"
                } 
	 }

    if ([path] =~ "FILEB")
        {
          grok{
		match => ["message","RECEIVED SOME MESSAGE, LINES IN THIS FILE ARE DIFFERENT THAN IN FILE 1, ID: %{NUMBERLC:tradeid}"]
               add_tag => [ "secondTaskStarted" ]
			}

		grok {
     		match => ["message","PUBLISING SOME MESSAGE, LINES IN THIS FILE ARE DIFFERENT THAN IN FILE 1, ID: %{NUMBERLC:tradeid}"]
			add_tag => [ "taskTerminated" ]
	  }
	elapsed
		{
			start_tag => "secondTaskStarted"
			end_tag => "taskTerminated"
			unique_id_field => "tradeid"
		}	
	}

	if ([path] =~ "FILEC")
    {
	grok {			
		match => ["message","RECEIVED ID: %{NUMBERLC:tradeid}"]
		add_tag => [ "taskAck" ]
	  }
	elapsed
	{
           start_tag => "taskTerminated"
           end_tag => "taskAck"
            unique_id_field => "tradeid"
	  }
	}
}

So let's assume that messages ("received"/"published") can appear in two files - FILEA & FILEB (but it doesn't have to, line can be stored only in one).
However, messages always are stored either in FILEA or FILEB, then after some while message is stored in FILEC.

FILE A\ 
         ----> FILE C
FILE B/ 

What I want to do is:
1] Count time which has passed using elapsed filter between firstTaskStarted & taskTerminated** < - IT WORKS**
2] Count time which has passed using elapsed filter between secondTaskStarted & taskTerminated <- IT WOKRS

3] Count time between taskTerminated stored in FILEA or FILEB and taskAck from FILEC.

And point 3 doesn't work for me, reason is quite obvious, elapsed plugin doesn't know what taskTerminated is as it comes from different elapsed region.

But How can I achieve that ?
Any ideas ?


(system) #2