Filebeat mongo module, grok pattern is incorrect

Elastic Stack Beats
1

The grok pattern ha an extra escape \ in it in the ingest pip line.

should look like this.

%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} %{WORD:mongodb.log.component} *\[%{WORD:mongodb.log.context}\] %{GREEDYDATA:mongodb.log.message}

Currently using.

ubuntu@ubuntu sudo apt list elasticsearch
Listing.
elasticsearch/stable 6.3.1 all [upgradable from: 6.3.0]

ubuntu@ubuntu:~$ sudo apt list filebeat
Listing.
filebeat/stable 6.3.1 amd64 [upgradable from: 6.3.0]

2

With your changes I get

2018-07-06T22:52:24.868+0200 ERROR pipeline/output.go:91 Failed to connect: Connection marked as failed because the onConnect callback failed: Error getting pipeline for fileset mongodb/log: Error JSON decoding the pipeline file: ingest/pipeline.json: invalid character '[' in string escape code

The \\[ escape is used in all the module pipelines, is there because the regular expression is inside a JSON string, so its really read by elasticsearch as \[.

Are you experiencing a problem parsing mongodb logs that got fixed by this change?

3

With the Current grok pattern, i'm getting a grok parse error for all logs coming from mongo 3.4. Beats is currently sending the message. But it is erroring out in the ingest pipeline.

error.message:Provided Grok expressions do not match field value: [2018-07-09T14:49:47.006+0000 I - [conn208899] Index Build (background): 250096500/463852976 53%]

This would probably be more proper for the 3 types of logs that get populated into mongo log.
%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} (-|%{WORD:mongodb.log.component}) *\\[%{WORD:mongodb.log.context}\\] %{GREEDYDATA:mongodb.log.message}

5

Managed to get this all to work by adding one more grok line to the patterns list.

"processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} %{WORD:mongodb.log.component} *\\[%{WORD:mongodb.log.context}\\] %{GREEDYDATA:mongodb.log.message}",
            "%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} - *\\[%{WORD:mongodb.log.context}\\] %{GREEDYDATA:mongodb.log.message}"
          ],
6

Thanks @cwray, now it makes more sense.

Can you open a Pull-Request to the beats repo or prefer if I do it for you?

7

Pull-request sent. It is stating that i did not sign the CLA. I went through the signing process before submitting the pull request. So I don't know what is going on there.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.