MongoDB Module - Grok Expression does not match

Elastic Stack Beats
1

The default grok expression created by filebeat for the ingest node pipeline does not match the logs sent to ES. Not using any sort of custom mongo logs, everything is just default.

Tried writing my own Grok expression, and I got it working via the simulate pipeline API, but it still wouldn't work when using Filebeat.

Filebeat v6.3.0
Mongo v3.4
ES v5.5 (AWS)

Running Mongo and Filebeat on an EC2 on AWS Linux

2

Hi @James_Quigley,

Could you paste some of the lines that are not being correctly parsed? This sounds like a bug but more info is needed to confirm so.

Best regards

3

2018-07-10T20:35:36.789+0000 I - [conn26567] end connection 10.6.4.66:59012 (46 connections now open)
2018-07-10T20:42:37.863+0000 I NETWORK [thread1] connection accepted from 10.6.4.44:56076 #26676 (42 connections now open)

This is how the lines look in the log file, when I send them via Postman, when they are in the filebeat logs, and when they appear in Elasticsearch in the error saying that grok failed to parse them

4

We've just received a contribution to fix this problem:

1 Like
5

I tried out the grok patterns in the PR, and unfortunately they still didn't work when using filebeat. (Though they did work via the simulate pipeline api)

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.