Pattern Matching

Elastic Stack Logstash
1

Using grok I want to parse a log file as you normally would but i also want to pull a specific piece of of the log record in a custom field. I have 2 patterns one is a nginx_combined with a x-forwarded field at the end. The second is the specific piece of the log that i'm trying to put into it own field. ? (?i)((http[s]?)(://.+?)/)

I've tried this but it isn't creating the custom field.

grok {
patterns_dir => ["/etc/logstash/patterns"]
break_on_match => false
match => { "message" => "%{SED_NGINX_COMBINE}" }
match => { "message" => "%{SED_HTTPHOST:httphost}" }
add_tag => ["drupal-staging"]
}

Basically I want both patterns to exist in a single record in elasticsearch. I first though that an addfield would work but is was suggested that i try it this way. Any ideas ?

2

Hello

You can do something like
match => { "message" => "<182>%{WORD:program}: %{COMBINEDAPACHELOG} \"((?<x_forwarded_for>%{IP:xff_clientip}, .*)|-)\" %{NUMBER:request_time:integer} %{IPORHOST:targethost}"}

Or using
=> "(%{SED_NGINX_COMBINE})|(%{SED_HTTPHOST:httphost}))"
Regards, Guillaume

3

with this method what does the '|' do ?

=> "(%{SED_NGINX_COMBINE})|(%{SED_HTTPHOST:httphost}))"

Does it cat them or is it a "or"

4

It's an or

5

Ok - unfortunately that's not what I'm trying to do. I'm trying to parse the same section of the log record 2 different ways and have both in the elasticsearch record as 2 different fields.

Particularly the referrer field. The first as %{QS:referrer} and then just part of it in a different field from this pattern: (?i)((http[s]?)(://.+?)/)

There doesn't seem to be a easy way to do this.

6

Hi,
Then maybe you should cheat, by example with something like
"message" => "%{GREEDYDATA:message2}"}
And then do two filter

  • on message with your first filter
  • on message2 (which is a clone of message) with the second one
7

ok i'll try that. Thanks

There isn't a reference somewhere for grok key words is there. I've been looking but can't seem to find one that lists and explains the pattern key words i see on git hub and other examples like your own. For example the WORD, QUOTEDSTRING, NUMBER, GREEDYDATA and many others ? I just would like to know how they are defined.

Thanks again

8

The definition of the patterns can be found here. The ones you mentioned are defined in this file.