Filebeat Ignoring Messages


(KJ) #1

Respected Folks,
Good day :slight_smile: May I please request your help w/ Filebeat conf.? I'm a newbie, so apologies in advance if I'm missing something.

My log file contains messages like,

<=- 20170414 12:58:39.768 PH2O incoming [135.46.166.47] << [135.46.166.47:7778] 3789 bytes
{"type":"eventEstablished","connection":"XXXXX","context":"ContextI",...}
-=>
<=- 20170414 07:04:49.889 PH2O incoming [135.46.166.59] << [135.46.166.59:7778] 2168 bytes
{"type":"eventReleased","connection":"XXXXX","context":"ContextI",...}
-=>
<=- 20170414 13:07:33.109 PH2O outgoing [135.46.166.47] >> [135.46.166.47:7778] 1886 bytes
{ "type": "requestMuteTransfer","connection":"XXXXX","context":"ContextI",...}
-=>
<=- 20170415 08:11:16.731 PH2O incoming [130.6.43.52] << [130.6.43.52:7776] 164 bytes
{ "context":"ContextI","sequence": 3,"type": "eventAuthenticated","content": { "loginid": "ABC123","authenticated": 1 } }
-=>
<=- 20170418 09:05:57.952 PH2O incoming [130.6.43.50] << [130.6.43.50:7778] 279 bytes
{"type":"eventError","connection":"XXXXX","context":"ContextI",...}
-=>
<=- 20170419 11:57:17.119 AH2O incoming [130.6.43.52] << [130.6.43.50:9977] 37 bytes
{"type":"pong","context":"reef2reef"}
-=>

What I'd like to retrieve through Filebeat?

  • only those lines with "eventEstablished", "eventReleased", "requestMuteTransfer", "eventError" AND each line immediately above as that contains the date/time

Here's my Filebeat Conf.
document_type: coral_reef
tags: ["json"]
exclude_lines: ["AH2O incoming", "AH2O outgoing", "MH2O incoming", "PluginInitiated", "Administration", "ping", "pong", "eventInfo", "requestRegisterAddress", "hello", "requestAuthenticate", "requestAnswerCall", "RequestOpenStatistic", "eventRegistered", "pong", "ping", "reef2reef"]
include_lines: ["PH2O incoming", "PH2O outgoing"]
multiline:
pattern: ^{[[:space:]]|"type":[[:space:]]|"(eventEstablished|eventReleased|requestMuteTransfer|requestCompleteTransfer|eventAuthenticated|eventAgentLogin|eventAgentLogout|requestAgentLogout|eventHeld|eventRetrieved|goodbye|requestReleaseCall|requestAgentNotReady|eventAgentNotReady|eventError)"
negate: false
match: after

What's happening?

  1. all messages with the text in "exclude_lines" are not getting published - GOOD
  2. messages with text in multiline pattern like "eventReleased", "requestCompleteTransfer" are also not getting published - NOT GOOD :frowning:
  3. messages that do not contain any of the text in multiline pattern are also getting published - NOT GOOD :frowning:

Questions,

  1. What multiline pattern settings do I set to explicitly match messages with the keywords "eventEstablished ,eventReleased ,requestMuteTransfer etc.) and append the line above?

Thanks in advance for your time, patience and cooperation :slight_smile: Wish you the very best.


(KJ) #2

Hello there!
Kindly request your thoughts/comments.

Thanks!


(ruflin) #3

Can you please format your config and code about above with 3 ticks ` before and after to make it readable and have the correct indentation?

Which filebeat version are you using?


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.