Filebeat Ignoring Messages

Respected Folks,
Good day :slight_smile: May I please request your help w/ Filebeat conf.? I'm a newbie, so apologies in advance if I'm missing something.

My log file contains messages like,

<=- 20170414 12:58:39.768 PH2O incoming [] << [] 3789 bytes
<=- 20170414 07:04:49.889 PH2O incoming [] << [] 2168 bytes
<=- 20170414 13:07:33.109 PH2O outgoing [] >> [] 1886 bytes
{ "type": "requestMuteTransfer","connection":"XXXXX","context":"ContextI",...}
<=- 20170415 08:11:16.731 PH2O incoming [] << [] 164 bytes
{ "context":"ContextI","sequence": 3,"type": "eventAuthenticated","content": { "loginid": "ABC123","authenticated": 1 } }
<=- 20170418 09:05:57.952 PH2O incoming [] << [] 279 bytes
<=- 20170419 11:57:17.119 AH2O incoming [] << [] 37 bytes

What I'd like to retrieve through Filebeat?

  • only those lines with "eventEstablished", "eventReleased", "requestMuteTransfer", "eventError" AND each line immediately above as that contains the date/time

Here's my Filebeat Conf.
document_type: coral_reef
tags: ["json"]
exclude_lines: ["AH2O incoming", "AH2O outgoing", "MH2O incoming", "PluginInitiated", "Administration", "ping", "pong", "eventInfo", "requestRegisterAddress", "hello", "requestAuthenticate", "requestAnswerCall", "RequestOpenStatistic", "eventRegistered", "pong", "ping", "reef2reef"]
include_lines: ["PH2O incoming", "PH2O outgoing"]
pattern: ^{[[:space:]]|"type":[[:space:]]|"(eventEstablished|eventReleased|requestMuteTransfer|requestCompleteTransfer|eventAuthenticated|eventAgentLogin|eventAgentLogout|requestAgentLogout|eventHeld|eventRetrieved|goodbye|requestReleaseCall|requestAgentNotReady|eventAgentNotReady|eventError)"
negate: false
match: after

What's happening?

  1. all messages with the text in "exclude_lines" are not getting published - GOOD
  2. messages with text in multiline pattern like "eventReleased", "requestCompleteTransfer" are also not getting published - NOT GOOD :frowning:
  3. messages that do not contain any of the text in multiline pattern are also getting published - NOT GOOD :frowning:


  1. What multiline pattern settings do I set to explicitly match messages with the keywords "eventEstablished ,eventReleased ,requestMuteTransfer etc.) and append the line above?

Thanks in advance for your time, patience and cooperation :slight_smile: Wish you the very best.

Hello there!
Kindly request your thoughts/comments.


Can you please format your config and code about above with 3 ticks ` before and after to make it readable and have the correct indentation?

Which filebeat version are you using?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.