Pattern Matching

Using grok I want to parse a log file as you normally would but i also want to pull a specific piece of of the log record in a custom field. I have 2 patterns one is a nginx_combined with a x-forwarded field at the end. The second is the specific piece of the log that i'm trying to put into it own field. ? (?i)((http[s]?)(://.+?)/)

I've tried this but it isn't creating the custom field.

grok {
patterns_dir => ["/etc/logstash/patterns"]
break_on_match => false
match => { "message" => "%{SED_NGINX_COMBINE}" }
match => { "message" => "%{SED_HTTPHOST:httphost}" }
add_tag => ["drupal-staging"]

Basically I want both patterns to exist in a single record in elasticsearch. I first though that an addfield would work but is was suggested that i try it this way. Any ideas ?


You can do something like
match => { "message" => "<182>%{WORD:program}: %{COMBINEDAPACHELOG} \"((?<x_forwarded_for>%{IP:xff_clientip}, .*)|-)\" %{NUMBER:request_time:integer} %{IPORHOST:targethost}"}

Or using
=> "(%{SED_NGINX_COMBINE})|(%{SED_HTTPHOST:httphost}))"
Regards, Guillaume

with this method what does the '|' do ?

=> "(%{SED_NGINX_COMBINE})|(%{SED_HTTPHOST:httphost}))"

Does it cat them or is it a "or"

It's an or

Ok - unfortunately that's not what I'm trying to do. I'm trying to parse the same section of the log record 2 different ways and have both in the elasticsearch record as 2 different fields.

Particularly the referrer field. The first as %{QS:referrer} and then just part of it in a different field from this pattern: (?i)((http[s]?)(://.+?)/)

There doesn't seem to be a easy way to do this.

Then maybe you should cheat, by example with something like
"message" => "%{GREEDYDATA:message2}"}
And then do two filter

  • on message with your first filter
  • on message2 (which is a clone of message) with the second one

ok i'll try that. Thanks

There isn't a reference somewhere for grok key words is there. I've been looking but can't seem to find one that lists and explains the pattern key words i see on git hub and other examples like your own. For example the WORD, QUOTEDSTRING, NUMBER, GREEDYDATA and many others ? I just would like to know how they are defined.

Thanks again

The definition of the patterns can be found here. The ones you mentioned are defined in this file.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.