Grok - Skipping Line

Elastic Stack Logstash
1

Good Evening Everyone,

I am a beginner in logstash. I was trying to parse a log file which have 159 lines in it. I am interested only in the first ,second and last line. Is there any way to skip or omit these lines with Grok ?

Is there any other logstash plugins allow me to do this?

2

Do the lines you want to skip have any characteristics (content-wise) that set them apart from the three first lines? Perhaps you can give an example of the input file.

(The grok filter itself won't help you to drop any events but the drop filter will.)

3

Magnus,

The only thing that differentiate is that the first and last lines have 3 #s
I am interested only in 1,6,and last line.
I need the Job Start Time, End time and Job Name[Move_File_job]
Please find the sample log attached:

Job Started at 2016/05/17 11:45:07 (jxxIx=20160427_171509_pxPwP, jxxxxxxutixxIx=20160517114507_txQxt)

[stxtistixs] xxxxxxtixx tx sxxxxt xx pxrt 10849
[stxtistixs] xxxxxxtxx

001:Move_File_job: xxxix(Prxxuxtixx) TxxxxxPix:20160517114507_txQxt RuxIx:xuxx
002:Jxx_xxxx_ix: Suxjxx hxs ixitixtx thx xx xpxx

001:xxxpxxxxtxxxsxs: xxxix(Prxxuxtixx)
002:xxxpxxxxtxxxsxs: Suxjxx hxs ixitixtx thx xx xpxx
Usixx xxxxxxx : prxxuxtixx

txxxtxxtxxxx_1 sxt xxy "xrxxxxSxrvixxxxxx" with vxxux "xxPxxxP.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxUsxrxxxx" with vxxux "xxx_xpxx"
txxxtxxtxxxx_1 sxt xxy "xrxxxxxxxitixxxxJxxxPxrxxxtxrs" with vxxux "xxxxuxtRxwPrxxxtxh=80000"
sxt xxy "xrxxxxxxxitixxxxJxxxPxrxxxtxrs" with vxxux "xxxxuxtRxwPrxxxtxh=80000"
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxhxxx" with vxxux " "
txxxtxxtxxxx_1 sxt xxy "xrxxxxHxst" with vxxux "xx01-sxxx.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxrvixxxxxx" with vxxux "xxPxxxP.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxPxsswxrx" with vxxux "xxrxxrt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxUsxrxxxx" with vxxux "ixtxrxrxtixx"
txxxtxxtxxxx_1 sxt xxy "xrxxxxUsxrxxxx" with vxxux "xxrxxrt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxhxxx" with vxxux " "
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxrvixxxxxx" with vxxux "xxRxxRT.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxPxrt" with vxxux "1587"
xrxxxxShxrxxxxxxxxtixx: xstxxxishixx shxrxx xxtxxxsx xxxxxxtixx xrxxxxxxxxpxxShxrxxxxxxxxtixx tx xxPxxxP.xxxxxx.xxx.xxxxxx.xxt xs xxx_xpxx
xrxxxxShxrxxxxxxxxtixx: xstxxxishixx shxrxx xxtxxxsx xxxxxxtixx xrxxxxxxxxxxxShxrxxxxxxxxtixx tx xxRxxRT.xxxxxx.xxx.xxxxxx.xxt xs xxrxxrt
xrxxxxShxrxxxxxxxxtixx: xstxxxishixx shxrxx xxtxxxsx xxxxxxtixx xrxxxxIxtxxrxtixxShxrxxxxxxxxtixx tx xxPxxxP.xxxxxx.xxx.xxxxxx.xxt xs ixtxrxrxtixx
xrxxxxShxrxxxxxxxxtixx: xstxxxishixx shxrxx xxtxxxsx xxxxxxtixx xrxxxxIxtxxrxtixxxxrShxrxxxxxxxxtixx tx xxRxxRT.xxxxxx.xxx.xxxxxx.xxt xs xxrxxrt
xrxxxxShxrxxxxxxxxtixx: xxxxxxtixx xstxxxishxx
xrxxxxShxrxxxxxxxxtixx: xxxxxxtixx xstxxxishxx
xrxxxxShxrxxxxxxxxtixx: xxxxxxtixx xstxxxishxx
xrxxxxShxrxxxxxxxxtixx: xxxxxxtixx xstxxxishxx
xxxxxtxxtRxxxxr: xx xxxtxxt whxrx xxxusx : whxrx xxxtxxtStr='Prxxuxtixx' xxx xxxtxxtxxxx='xRxxxx_xxx'
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxrvixxxxxx" with vxxux "xxPxxxP.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxUsxrxxxx" with vxxux "xxx_xpxx"
txxxtxxtxxxx_1 sxt xxy "xrxxxxxxxitixxxxJxxxPxrxxxtxrs" with vxxux "xxxxuxtRxwPrxxxtxh=80000"
txxxtxxtxxxx_1 sxt xxy "xrxxxxHxst" with vxxux "xx01-sxxx.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxPxsswxrx" with vxxux "xxxxxx"
txxxtxxtxxxx_1 sxt xxy "xrxxxxPxrt" with vxxux "1587"
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxhxxx" with vxxux " "
xrxxxxShxrxxxxxxxxtixx: xstxxxishixx shxrxx xxtxxxsx xxxxxxtixx xrxxxxxxxxxShxrxxxxxxxxtixx tx xxPxxxP.xxxxxx.xxx.xxxxxx.xxt xs xxx_xpxx
xrxxxxShxrxxxxxxxxtixx: xxxxxxtixx xstxxxishxx

txxxtxxtxxxx_1 sxt xxy "xrxxxxxxxitixxxxJxxxPxrxxxtxrs" with vxxux "xxxxuxtRxwPrxxxtxh=80000"
txxxtxxtxxxx_1 sxt xxy "xrxxxxHxst" with vxxux "xx01-sxxx.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxPxsswxrx" with vxxux "xPIzhxxISP"
txxxtxxtxxxx_1 sxt xxy "xrxxxxPxrt" with vxxux "1587"
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxhxxx" with vxxux " "
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxrvixxxxxx" with vxxux "xxPxxxP.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxUsxrxxxx" with vxxux "xxx_xUxIT"

xrxxxxShxrxxxxxxxxtixx: xstxxxishixx shxrxx xxtxxxsx xxxxxxtixx xrxxxxxxxxuxitShxrxxxxxxxxtixx tx xxPxxxP.xxxxxx.xxx.xxxxxx.xxt xs xxx_xUxIT
xrxxxxShxrxxxxxxxxtixx: xxxxxxtixx xstxxxishxx
xxxxxtxxtRxxxxr: xx xxxtxxt whxrx xxxusx : whxrx xxxtxxtStr='Prxxuxtixx' xxx xxxtxxtxxxx='xRxxxx_IxTxxRxTIxx_xxStxxx'
txxxtxxtxxxx_1 sxt xxy "xrxxxxxxxitixxxxJxxxPxrxxxtxrs" with vxxux "xxxxuxtRxwPrxxxtxh=80000"
txxxtxxtxxxx_1 sxt xxy "xrxxxxHxst" with vxxux "xx01-sxxx.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxPxsswxrx" with vxxux "xxrxxrt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxPxrt" with vxxux "1587"
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxhxxx" with vxxux " "
txxxtxxtxxxx_1 sxt xxy "xrxxxxSxrvixxxxxx" with vxxux "xxxxxRT.xxxxxx.xxx.xxxxxx.xxt"
txxxtxxtxxxx_1 sxt xxy "xrxxxxUsxrxxxx" with vxxux "xxrxxrt"
xrxxxxShxrxxxxxxxxtixx: xstxxxishixx shxrxx xxtxxxsx xxxxxxtixx xrxxxxxxStxxxShxrxxxxxxxxtixx t
001:xs_Vxixxxsx_IUx: xxxix(Prxxuxtixx) TxxxxxPix:20160517114507_txQxt RuxIx:xuxx

Job Ended Successfuxxy xt 2016/05/17 11:46:04 (jxxIx=20160427_171509_pxPwP, jxxxxxxutixxIx=20160517114507_txQxt)

4

This would've been much easier to grasp if you had formatted the log as code so that the "###" lines weren't formatted as section headers.

Use a conditional to drop all lines not matching one of the patterns that you're interested in. Something like this should work:

filter {
  if [message] !~ /^(Job Started at|Job Ended |001: Move_File_Job: )/ {
    drop { }
  }
}
5

Cool. Now i can drop the lines i wanted. Now these 3 lines are parsed as 3 different events.
Can i use grok just below that if i need to extract the Start time, End time and Job name?

6

Sure. If you want start and end time (etc) in a single event the aggregate filter should be useful.

7

Thank you Magnus.

But Aggregate is giving me error..

8

The very first paragraph of the aggregate filter documentation tells you how to install the plugin.

10

Thank You Magnus