Extract a timestamp between square brackets [SOLVED]

[dr01]

I need to extract the timestamp from this line:

[Tue, 07 Jun 2018 15:53:29 +0200] codedeploy-agent started 

1)) I've tried the following filter:

mutate {
   copy => { "message" => "ts" }
   gsub => [ "ts", "\[", "" ]
   gsub => [ "ts", "\].*", "" ]
}

but it has no effect: ts is identical to message.

2)) If I don't escape the square brackets and use the literal strings...

mutate {
   copy => { "message" => "ts" }
   gsub => [ "ts", "[", "" ]
   gsub => [ "ts", "] codedeploy-agent started", "" ]
}

... the system throws an error [ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<RegexpError: premature end of char-class: /[/>

3)) If I escape just the square brackets and use the literal string instead of the regex...

mutate {
   copy => { "message" => "ts" }
   gsub => [ "ts", "\[", "" ]
   gsub => [ "ts", "\] codedeploy-agent started", "" ]
}

ts is identical to message, again.

4)) This filter:

dissect {
   mapping => { "message" => "\[%{ts} %{+ts} %{+ts} %{+ts} %{+ts} %{+ts}\] codedeploy-agent started" }
}

returns a ts which, strangely, contains ue, 07 Jun 2018 15:53:29 +0200] codedeploy-agent started.

5)) Without escaping the square brackets:

dissect {
   mapping => { "message" => "[%{ts} %{+ts} %{+ts} %{+ts} %{+ts} %{+ts}] codedeploy-agent started" }
}

I get a ts which is again identical to message.

There's probably an obvious solution, but what is it?

[Badger]

mutate {
   copy => { "message" => "ts" }
   gsub => [ "ts", "\[", "" ]
   gsub => [ "ts", "\].*", "" ]
}

Mutate does operations in a fixed order, and it looks like gsub comes before copy, so split this into two mutates.

    mutate {
        copy => { "message" => "ts" }
    }
    mutate {
        gsub => [ "ts", "\[", "" ]
        gsub => [ "ts", "\].*", "" ]
    }

[dr01]

Many thanks, it worked. It's very misleading that the mutate filter follows its own order to apply operations!

If there are two or more mutate filters, are they always applied in the order they're written? What about other plugins?

[Badger]

If they are written as separate filters they are applied in the order written.

The other case I can think of where order is ignored is the "common options" that most filters support. These are applied after the filter executes, and are conditional upon the filter working. Thus, for a date filter, you can remove_field the text field that contains the date, and if you get a _dateparsefailure the field will be left on the event, so that you can see what it failed to parse.