How to Parse Interleaved Multi-Service Logs with Filebeat/Logstash - Extracting Complete Transaction Messages

Elastic Stack Beats
1

Hi Guys,
I'm new on FileBeat or Logstash, here is my problem,
I need to parse log files where multiple applications alternately write entries into the same file using Filebeat or Logstash.

Our scenario:

  • Multiple services write to the same log file with entries like:
[Service1] Txn started: AAA Txn
[Service1] bla bla bla 
[Service2] Txn started: BBB Txn
[Service2] bla bla bla
[Service2] Txn finished: BBB Txn
[Service1] Txn finished: AAA Txn
  • Each service (Service1/Service2) only has one active transaction at a time
  • Goal: Extract complete transaction logs from start to finish as separate messages for each service like
    message 1:
[Service1] Txn started: AAA Txn\n[Service1] bla bla bla\n[Service1] Txn finished: AAA Txn

and 2

[Service2] Txn started: BBB Txn\n[Service2] bla bla bla\n[Service2] Txn finished: BBB Txn