MongoDB Module - Grok Expression does not match

[James_Quigley]

The default grok expression created by filebeat for the ingest node pipeline does not match the logs sent to ES. Not using any sort of custom mongo logs, everything is just default.

Tried writing my own Grok expression, and I got it working via the simulate pipeline API, but it still wouldn't work when using Filebeat.

Filebeat v6.3.0
Mongo v3.4
ES v5.5 (AWS)

Running Mongo and Filebeat on an EC2 on AWS Linux

[exekias (Carlos Pérez Aradros)]

Hi @James_Quigley,

Could you paste some of the lines that are not being correctly parsed? This sounds like a bug but more info is needed to confirm so.

Best regards

[James_Quigley]

2018-07-10T20:35:36.789+0000 I - [conn26567] end connection 10.6.4.66:59012 (46 connections now open)
2018-07-10T20:42:37.863+0000 I NETWORK [thread1] connection accepted from 10.6.4.44:56076 #26676 (42 connections now open)

This is how the lines look in the log file, when I send them via Postman, when they are in the filebeat logs, and when they appear in Elasticsearch in the error saying that grok failed to parse them

[adrisr (Adrian Serrano)]

We've just received a contribution to fix this problem:

Adding another grok pattern to the filebeats mongo module pipeline.

[James_Quigley]

I tried out the grok patterns in the PR, and unfortunately they still didn't work when using filebeat. (Though they did work via the simulate pipeline api)