Merge the varnish logs into a single line

[Yuvraj_S]

hi team,

i am using varnish application which gives a lots of logs in the messages. The need is to merge these lines into a single line in the message. Can you please suggest any options of merging those lines.

actual logs:

• << BeReq >>

• Begin
• Timestamp
• BereqURL
• BereqProtocol
• BereqHeader
• BereqHeader
• BereqHeader
• BereqHeader
• BereqHeader
• BereqHeader
  END

expected logs in the ELK,

• << BeReq >> - Begin - Timestamp - BereqURL - BereqProtocol - BereqHeader - BereqHeader - BereqHeader - BereqHeader - BereqHeader - BereqHeader END

need to use a string matching in the grok filter which will start merging the logs when it find the string "BeReq" and merge till it finds the string "END"

[Badger]

You could do this using a multiline codec on your input.

[Yuvraj_S]

Hi Thanks for the reply. But I saw it in multiline, it just allows me to club 2 lines at a time, which is not what i want.

here is how my log looks like,
#cat test.log
rob
adam
ram
shyam
amanda
bob
silva
adam
d
e
f
paks

Output which is coming currently:

adam
ram
#############
the next line comes
....
...
then,
adam
d
################
the next line starts again.

actual output which i require is,

adam
ram
shyam
amanda
bob
silva

merge the lines when it meets adam till it finds next adam,

here is my config how it looks,

multiline.pattern: ^adam
multiline.negate: false
multiline.match: before

adding the info on the architecture am using,

using filbeat in client and then ship the logs to logstash,

[Badger]

codec => multiline { pattern => "^adam" negate => true what => previous auto_flush_interval => 1 } }

will get you

   "message" => "rob",
   "message" => "adam\nram\nshyam\namanda\nbob\nsilva",
   "message" => "adam\nd\ne\nf\npaks",

[Yuvraj_S]

Thanks Badger, tried the same and it was working