Aggregate function help

[Ameeruddin_Mohammed]

hi,
i have logs in this format and i want to start the aggregation when start comes in the line and end the aggregation when end occurs. the aggregation is based on "username".
i was able to use aggregate function but it does not split the documents.
username;command;start/end
user1;command1;
user1;command2;start
user2;command3;start
user3;command4;
user1;command5;
user2;command6;
user3;command7;start
user2;command8;end
user1;command9;
user2;command10;end
user3;command11;end
user2;command12;
user1;command13;end
user2;command14;
user1;command15;
user3;command16;

needed
doc1
user1;command1;

doc2
user1;command2;start
user1;command5;
user1;command9;
user1;command13;end

doc3
user1;command15;

doc4
user2;command3;start
user2;command6;
user2;command8;end

doc5
user2;command12;
user2;command14;

doc6
user3;command4;

doc7
user3;command7;start
user3;command11;

doc8
user3;command16;

[Badger]

You want command12 and command14 aggregated even though there is no start/end, but you do not want command1 aggregated? So until a start appears for a user you do not want aggregation, but once a start has been seen you want everything aggregated?

[Ameeruddin_Mohammed]

Hi,
Sorry my bad, those two should be separate documents.
Yes, all documents should be individually captured. Once start starts aggregation should happen and aggregation should end once end is found.
Post end till new start per user no aggregation.

[Badger]

I tried doing this with three aggregate filters (similar to example 1 in the documentation) and could not get it to work. I then decided to do it in ruby.

     csv {
        columns => [ "[@metadata][user]", "[@metadata][command]", "[@metadata][startEnd]" ]
        separator => ";"
    }

    ruby {
        code => '
            @users ||= {}

            meta = event.get("@metadata")
            user = meta["user"]
            @users[user] ||= {}
            msg = event.get("message")

            if meta["startEnd"] == "start/end"
                event.cancel
            elsif meta["startEnd"] == "start"
                @users[user] = { "started" => true, "commands" => [ msg ] }
                event.cancel
            elsif meta["startEnd"] == "end"
                @users[user]["commands"] << msg
                event.set("commands", @users[user]["commands"])
                @users[user].delete("started")
            else
                if @users[user].has_key?("started")
                    @users[user]["commands"] << msg
                    event.cancel
                else
                    event.set("commands", [ msg ])
                end
            end
        '
    }

will do the wrong thing if the start/end messages are not paired up, but it's close.