Regular Expression not working

Elastic Stack Logstash
1

Hi,

I am BEGINNER at Logstash 2.1.
Trying to write config file for below log.

LogFile:

Timestamp Process TID Area Category EventID Level Message Correlation
05/29/2017 07:24:00.28 wsstracing.exe (0x096C) 0x0BC4 SharePoint Foundation
Unified Logging Service b9wt High Log retention limit reached. 48f5d7e0-d67d-4b6a-99c6-701385ae9636
05/29/2017 07:24:00.28 wsstracing.exe (0x096C) 0x0BC4 SharePoint Foundation Tracing Controller Service 8096 Information Usage log retention limit reached. Some old usage log files have been deleted. 48f5d7e0-d67d-4b6a-99c6-701385ae9636
05/29/2017 07:24:00.53 OWSTIMER.EXE (0x03BC) 0x1A5C SharePoint Foundation Timer 5utp Verbose Scheduled timer job Diagnostic Data Provider: SQL Blocking Queries id {06DDAEBE-C8DB-44BA-9F48-671740732C29} 48f5d7e0-d67d-4b6a-99c6-701385ae9636
05/29/2017 07:24:00.53 OWSTIMER.EXE (0x03BC) 0x1A5C SharePoint Foundation Timer 5utp Verbose Scheduled timer job Config Refresh id {ADC8F1FF-BE72-4BC9-AB4B-37D2C67AFE21} 48f5d7e0-d67d-4b6a-99c6-701385ae9636
05/29/2017 07:24:00.53 OWSTIMER.EXE (0x03BC) 0x1A5C SharePoint Foundation Timer 5utp Verbose Scheduled timer job Diagnostic Data Provider 48f5d7e0-d67d-4b6a-99c6-701385ae9636
05/29/2017 07:24:00.53 OWSTIMER.EXE (0x03BC) 0x068C SharePoint Foundation Timer 8e45 Verbose Begin invoke timer job Config Refresh 48f5d7e0-d67d-4b6a-99c6-701385ae9636

Config File:

filter
{
if [type] == "logs" {
grok{
match=>["message",""]
overwrite => ["message"]
}
date{
match => ["timestamp","MM/dd/yyyy HH:mm:ss"]
remove_field=>["timestamp"]
}
}
}

I have a couple of below questions. Please help.

1. Need to define a GROK PATTERN for above mentioned log. I need all parameters' value specified in header of logfile.

2. How to skip headers in log file during logstash parsing.

Kindly help asap.

2

Have you tried using the grok constructor web site to get help creating your grok expression?

3

Hi Magnus,

Thanks for your time, man!

I was trying to construct using that website.
I was unable to create for below.

When parameter value contains a couple of space, then how can I differentiate it..?

Ex.
TID /MultipleSpace/ Area /MultipleSpace/ Category /MultipleSpace/ EventID /MultipleSpace/ Level

"0x0BC4" /MultipleSpace/ "SharePoint Foundation" /MultipleSpace/ "Unified Logging Service" /MultipleSpace/ "b9wt" /MultipleSpace/ "High"

Also I want to ignore headers (very first line of log file) during parsing. Do I need to write anything in filter tag..?

4

\s means "any whitespace character", + means "one or more of the preceding token", and so \s+ means "one or more whitespace characters".

Also I want to ignore headers (very first line of log file) during parsing. Do I need to write anything in filter tag..?

If only the header and no actual log lines begin with "Timestamp" you could e.g. say

if [message] =~ /^Timestamp / {
  drop { }
}

to drop events that begin with "Timestamp ".

5

Thanks Magnus for your prompt response ..!

It works.
Now I started to get the things about logstash.

Can you please let me know how can I create and run my own regex patterns? Will Logstash allow that..?

6

Can you please let me know how can I create and run my own regex patterns?

The grok filter documentation has a rather long section about custom patterns.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.