Grok formatting help

Elastic Stack Logstash
1

Just getting started with elastic this is my second attempt now that I have some free time I'm getting somewhere with it. Having some trouble finding the correct way to format my grok to correctly grab the fields.

Main Goals
I'd like to correctly format the date to iso8601
Grab the teamviewer id which varies from 9 to 10 digits "my ^\d{9.10}$ regex doesn't seem to work"

Here's some sample log data I'm looking at.

2019/05/02 07:38:05.431   904  5468 G1   CMainWindow::ShouldShowDynamicPassword(): dynPw=1 allowIncoming=1 id=0123456789 ka=0 lanAllowed=0 lanOnly=0 networkState=1 showDynPwd=
2019/05/02 07:38:06.134  2796  3772 S0!!!Connect to Master master5.teamviewer.com / 185.188.32.5 failed!
2019/05/02 07:38:06.134  2796  3772 S0!! CMasterConnectorAsio::HandleMasterResponseLogin(): MasterConnect failed. ErrorCode=10
2019/05/02 07:38:09.134  2796  3772 S0   Activating Router carrier
2019/05/02 10:51:00.083  2404  5904 D1!! CGrabMethodGDI::Grab: Fullscreen grab with BitBlt failed with code 0
2019/05/02 11:13:19.469  2156  2272 S0   Negotiating session encryption: client hello received from 123456789, RSA key length = 2048
2019/05/02 11:14:20.188  2156  2272 S0   CGatewaySession[44]::EndSession(): Session to 123456789 ended. Estimated capacity=83294kBit/s, Latency=1ms
2019/05/02 11:40:18.516  8740  3408 H64  explorer.exe: ResumeAllThreads: resumed 17 threads, max count 17
2

To the first line you can do something like this using grok filter:

%{YEAR:year}/%{MONTHNUM:Month}/%{MONTHDAY:Day} %{TIME:Time}%{GREEDYDATA}(?<id>id=%{NUMBER})

I do not know if you've seen https://grokdebug.herokuapp.com/
Check it out if you want better tuning your own filter

3

Alternatively

    grok { match => { "message" => "^(?<[@metadata][timestamp]>.{23})\s+(?<id>\d{3,4}  \d{4}) %{GREEDYDATA:restOfLine}" } }
    date { match => [ "[@metadata][timestamp]", "YYYY/MM/dd HH:mm:ss.SSS" ] }

There are many ways to get what you want...

6

Yeah I had something close to this problem is the log varies so much what works on one line does not on the other. I was using the online debugger but ended up finding the one in kibana under dev tools.

7

Got it close but I just need to figure out how to ignore part of the filter if it doesn't exist

%{YEAR:year}/%{MONTHNUM:month}/%{MONTHDAY:day}%{SPACE}%{TIME:time}%{SPACE}%{INFO:id1}%{SPACE}%{INFO:id2}%{SPACE}%{GREEDYDATA:messagestart}%{SPACE}%{GREEDYDATA}%{SPACE}(?<teamviewerid>%{TEAMVIEWER_ID})%{GREEDYDATA:messageend}

Custom pattern
TEAMVIEWER_ID \d{9,10}(?!0)
INFO \d{3,4}(?!0)

8

I guess you can use greedydata and leave as unnamed data. Because the default configuration only parse named data.