Logstash elapsed filter plugin: can't connect start and end events

Elastic Stack Logstash
1

Hi.
Currently, my logstash filter looks something like this:

filter {
  if ('text1' in [message]) {
    grok {
      ...
      add_tag => [ "connection_established" ]
    }

    aggregate {
      ...
    }

    elapsed {
      start_tag => "connection_established"
      end_tag => "connection_delete"
      unique_id_field => "conn_fingerprint"        
    }
  }
  else if ('text2' in [message]) {
    grok {
      ...
      add_tag => [ "connection_delete" ]
    }

    aggregate {
      ...
    }

    elapsed {
      start_tag => "connection_established"
      end_tag => "connection_delete"
      unique_id_field => "conn_fingerprint"
      add_field => {
          "connection_duration" => "%{elapsed_time}"
      }
    }
  }
}

But, checking results, I see that elapsed plugin is not working correctly - for every "end event" (events with tag connection_delete) I get tag elapsed_end_without_start. (edit: of course, I imply "start events" are present, i.e. are present and are parsed correctly the same way as "end events", and "start event" 's timestamps are earlies than "end events". unique_id_field value is also equal for related start and end events).

I've experimented around a bit (my pipeline.workers was already 1 since I use aggregate plugin; also tried to set pipeline.batch.size to 1 - no difference), but haven't reach any progress.

My main question is: can I use elapsed plugin like that? (when it's configuration is defined in two different branches of logstash config)
Or, maybe, it is strongly required that elapsed configuration is mentioned only once?

2

No, you cannot. The filter stores state in an instance variable, so the instance in one branch will know nothing about what the instance in the other branch has seen.

3

Thanks, experimented around and can confirm that "elapsed" works fine if I use it in one branch only.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.