Aggregate filter plugin output in a single document - Logstash

Elastic Stack Logstash
1

Hi guys, I'm using an aggregate filter plugin to map certain information to the main document that I'm processing with logstash but I don't understand why at the end I'm getting 2 documents instead of 1, the first of them generated by the all the plugins in the filter and another exclusively for the aggregation plugin, which by the way also is in the filter section. I'll put the output that I'm reciving below.

Is there a way to obtain all in one single document? I'd really appreciate your suggestions.

{
                 "transactionid" => "414d51204553422e4e312e4d5120202061b7b99e28dc0341",
                        "inicio" => "17:35:56.330",
                    "statusDesc" => [
        [0] "transaction ok"
    ],
              "msg_elapsed_time" => 15.278,
                         "channel => "ABC",
                   "ServiceOper" => "AccountEndorsementNoveltyAdd.setEndorsementNovelty.900172.",
                   "ServiceName" => "AccountEndorsementNoveltyAdd",
                     "statussys" => [
        [0] "0"
    ],
                    "final_time" => 2022-01-01T22:36:11.608Z,
                    "@timestamp" => 2022-01-17T15:49:41.605Z,
                "message_status" => "ok",
    "message_status_description" => "transaction ok",
                         "reqid" => "623e7c95-8669-4fa0-87cd-2cfafe23b491",
                     "Operation" => "setEndorsementNovelty",
                         "final" => "17:36:11.608",
                    "start_time" => 2022-01-01T22:35:56.330Z,
                      "CTRFMGID" => "900172.",
                          "Date" => "11-01-2022"
}
{
                 "tx_service" => "AccountEndorsementNoveltyAdd",
              "transactionid" => "414d51204553422e4e312e4d5120202061b7b99e28dc0341",
                "tx_end_time" => "17:36:11.608",
          "tx_total_messages" => 1,
                    "tx_date" => "11-01-2022",
                 "@timestamp" => 2022-01-17T15:50:00.743Z,
                  "tx_status" => "ok",
              "tx_start_time" => "17:35:56.330",
       "tx_service_operation" => "setEndorsementNovelty",
                   "tx_canal" => "ABC",
    "tx_more_than_1_messages" => true,
            "tx_elapsed_time" => 15.278,
                       "tags" => [
        [0] "_aggregatetimeout"
    ],
                      "tx_id" => "414d51204553422e4e312e4d5120202061b7b99e28dc0341"
}
2

An aggregate filter is just another filter. Every event that goes through an aggregate will get sent through the rest of the pipeline and on to the outputs. Just as they would for a mutate filter.

If you do not want the unaggregated events to continue through the pipeline you can call event.cancel in the code option.

In addition to the unaggregated events the aggregate filter will create a new event every time a timeout occurs if push_map_as_event_on_timeout is set, and a new event every time the task_id changes if push_previous_map_as_event is set.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.