[Need Help] Multiple pattern in one log file

Elastic Stack Logstash
1

I am trying to send xtreemfs log files to logstash and parse them into elastic search

A problem with xtreemfs log files is that they begin with some thing like

########### Configuration ###########
# Stuff
# More Stuff
# More stuff
# Even more stuff
### End of configuration stuff

[ I | Data | Data | data | Data ] Message
[ I | Data | Data | data | Data ] Message
[ I | Data | Data | data | Data ] Message
[ I | Data | Data | data | Data ] Message
[ I | Data | Data | data | Data ] Message

As you can see, two different patterns exist in one log file.

Is there a way for me to group the line beginning with # together in one single event? And then parse the rest of logs beginning with [ using another grok?
What is the proper way to handle this? I am using logstash 1.4.2.
Is parsing some thing like this well supported in logstash?

2

If the lines begin in #, you can do a conditional and then drop them entirely.

3

I don't want to drop them.

I still want to group them together and show them as one event.

4

Then you will want to do a conditional but with a mutliline filter/codec.

5

Hey Mark! Thanks for the great help.

I was able to group the # logs together with this codec under file input.

codec => multiline {
  pattern => "^\["
  negate => true
  what => previous
}

Now the multiline events are nicely tagged with "multiline" and allow me to do further processing if needed!