Aggregate Plugin push_previous_map_as_event

[Guanda_Li]

I'm aggregating streaming connection log from nginx server. Usually there are several similar events coming with same id, then there are another bunch of similar events with another id coming. I'm trying to get the number of consecutive events with same id.

The problem is that there is no start and end event in my case, every time I see a new id it should be the end of previous map. I used push_previous_map_as_event and it works fine, but it keeps the previous map alive and never end it. Is there any way I can set end_of_task => true after push_previous_map_as_event is true?

Thank you!

[Badger]

You could switch to push_map_as_event_on_timeout with a short timeout.

[Guanda_Li]

I just tried it out, but push_map_as_event_on_timeout also keeps the previous map alive.

For example, I have 10 events with id 1 coming, and then I have 10 events with id 2 coming. Then I see 10 events with id 1 again.

What I want is:

id: 1
counter: 10

id: 2
counter: 10

id: 1
counter: 10

But what happened is:

id: 1
counter: 10

id: 2
counter: 10

id: 1
counter: 20

id: 2
counter: 10

So it not only keeps the previous map alive but also push all of them out after I had a new aggregation task coming.

[Badger]

Try

        push_map_as_event_on_timeout => true
        push_previous_map_as_event => true
        timeout => 0

[Guanda_Li]

Thank you for your help! Actually push_map_as_event_on_timeout works good. The reason I don't see it working correct is because I am manually pasting some log as input. And maybe that's not really a streaming.

I just verified the number using real streaming data and it looks good. Thank you for your help!