Multiline negation

Elastic Stack Logstash
1

This is the data I have (this is one entry each).

"MAIN_cache_hit" => 1472612747
"MAIN_client_req" => 1524049030
"MAIN_fetch_1xx" => 0
"MAIN_fetch_204" => 0

I'm trying to group MAIN_cache_hit and MAIN_client_req on a single logstash entry and leave the rest how they are. However this is grouping all of the fields together instead of creating one event with two fields and then one event per field. What am I doing wrong?

    multiline {
        pattern => "^MAIN_(cache|client)_(\w+)\z"
        negate => "true"
        what => "next"
    }
2

I think you're better off using the aggregate filter.

3

I don't understand why the multiline isn't working. This is what my data looks like:

Data

    multiline {
        pattern => "^MAIN"
        what => "next"
    }

I just want to merge all events together that start with MAIN since I want to do math operations between them and I can't if they're separate events.

4

What does your input plugin (where you use the multiline filter) look like? Codecs receive a payload in the form of a string and applies the pattern to that string. In the examples of events you showed (for some odd reason as a screenshot; please use copy/paste next time) it's not clear what the original payload looks like.

5

I'm using the multiline plugin in the filter not on the input. The input looks like this:

input {
    redis {
        host => "127.0.0.1"
        port => "6379"
        data_type => "list"
        key => "filebeat"
    }
}

I posted the data as a screenshot because I didn't want to create a long post. And here is the data I receive (there's two types of them, I want to merge one type together to operate between events)

{
    "@timestamp" => "2016-07-15T05:52:40.402Z",
          "type" => "vc_server",
     "parameter" => "/diario/hoy/um/sumariorss_xml",
         "value" => 5,
      "hostname" => "agslx-hpclavc05"
}
{
    "@timestamp" => "2016-07-15T05:52:40.402Z",
          "type" => "vc_server",
     "parameter" => "/destacados/externos/dx_zonales_2_html",
         "value" => 4,
      "hostname" => "agslx-hpclavc05"
}
{
         "@timestamp" => "2016-07-15T05:52:40.402Z",
               "type" => "vc_server",
           "hostname" => "agslx-hpclavc05",
    "MAIN_client_req" => 1695530932,
               "tags" => [
        [0] "varnishstat"
    ]
}
{
        "@timestamp" => "2016-07-15T05:52:40.402Z",
              "type" => "vc_server",
          "hostname" => "agslx-hpclavc05",
    "MAIN_cache_hit" => 1637393101,
              "tags" => [
        [0] "varnishstat"
    ]
}

The goal is to group the MAIN one togethers since I want to divide MAIN_cache_hit by MAIN_client_req.

6

How do you know that the two MAIN events will always be consecutive with nothing else inbetween?

Either way, the multiline filter acts on the values of fields. I don't think you can get it to merge events based on the presence of certain fields.

Again, I think you should use the aggregate filter.

7

You're completely right, I was thinking of how I'm setting it up right now and I know the order of the logs I'm sending from one server but when I start scaling horizontally then there's no way to determine the two events will arrive together. I'll look into the aggregate filter and see how can I do it.