Representing text literals in match statement

Elastic Stack Logstash
1

Guys,

I'm working on a filter to match this log entry:

491 <14>1 2015-10-20T20:17:42.968706+00:00 loggregator 74cb190b-7ae2-4f25-8954-c18a0fd3ca37 [App/0] - -   \u001b[1m\u001b[36mSQL (0.9ms)\u001b[0m  \u001b[1mUPDATE `delayed_jobs` SET `delayed_jobs`.`locked_at` = '2015-10-20 20:17:42', `delayed_jobs`.`locked_by` = 'host:18um6m02fhc pid:29' WHERE ((run_at <= '2015-10-20 20:17:42.966763' AND (locked_at IS NULL OR locked_at < '2015-10-20 16:17:42.966811') OR locked_by = 'host:18um6m02fhc pid:29') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1\u001b[0m

I've added a bunch of statements to parse the above text. Please bear in mind what I'm about to show you is FAR from complete and only about 1/2 way done:

match => ['message', '%{BASE10NUM:491} %{SPACE} %{SYSLOG5424PRINTASCII:<14>1}  %{SYSLOG5424PRINTASCII:2015-10-20T20:17:42.968706+00:00} %{SPACE} %{HOST:loggregator} %{SPACE} %{JAVAFILE:74cb190b-7ae2-4f25-8954-c18a0fd3ca37} %{SPACE}  ]
        }

Where I'm getting held up, is in representing text that will be the same for every log entry, This text exists in every log entry coming into LS for this type of log:

[App/0] - -

Sorry if this is a simple question. But how can I represent that in my match statement so that LS will expect exactly that text pattern every time? Do I have to escape it, quote it, etc?

Thanks

2

%{BASE10NUM:491} %{SPACE} %{SYSLOG5424PRINTASCII:<14>1}

You seem to be misunderstanding the %{pattern:fieldname} grok syntax. The pattern identifies how you want to match the input text while the fieldname is the name of the field in which to store the matching text. Unless you want to store the first integer into a field named 491 the expression above doesn't make sense.

Also, since you have spaces on both sides of %{SPACE} you're effectively trying to match three spaces (well, technically a space followed by any whitespace character followed by another space).

But how can I represent that in my match statement so that LS will expect exactly that text pattern every time? Do I have to escape it, quote it, etc?

In regular expressions most characters lack special meaning and thus require no escaping. Square brackets do have a special meaning though, so to match "[App/0] - - " you need "\[App/0\] - -".

3

ok thanks for clearing that up about needing only to escape brackets.

As to mistaking the meaning %{pattern:fieldname} , I certainly don't blame you for thinking that. The reason I had that setup that way was because as I was builing the grok pattern, I was putting the original text in the fieldname so that I could remember what I was matching. I actually understand that you're supposed to give the fieldname an arbirratry name you can use to identify it.

Still new to building grok patterns, so that's why I was doing it that way. Sorry for the confusion!