FIlebeat multiline pattern for Elasticsearch logs?

Elastic Stack Beats
1

I managed to puzzle out a multiline filter for Filebeat to parse PostgreSQL CSV replication logs, but I find the docs on multiline pretty lacking in examples. Right now I'm trying to figure out a pattern for merging error messages in the Elasticsearch logs themselves, which I'm ingesting into an ELK stack. Has anyone else already figured this one out? If so I'd appreciate it if you could share the pattern. Thanks in advance!

2

Something like

multiline:
  pattern: <some pattern that match an ES log timestamp>
  negate: true
  match: after

should work, no?

3

@jeffkirk1any input for multiline is welcome. I don't have an elasticsearch log with stack trace lying around right now, can you please add one?

As we're collecting some multiline patterns/scenarios. Your use case sounds very interesting. Can you share your pattern + some log output with us?

I got a small 'script' to play with regexes as used in multiline feature: http://play.golang.org/p/ABEJaX_lsK
Update pattern/negate/content as required. With sample data every line beginning with true will be merged into multiline event.

4

I will be adding an example pattern in the next few days.

5

Here's the FIlebeat multiline pattern for joining Elasticsearch logs Java messages that worked for me:

multiline:
pattern: '[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}]'
match: after
negate: true