# Filebeat mongo module, grok pattern is incorrect

**URL:** <https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971>\
**Category:** Beats\
**Created:** [July 6, 2018, 6:00pm UTC](https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971 "2018-07-06T18:00:39Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![cwray](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@cwray](https://discuss.elastic.co/u/cwray)\
**Post date:** [July 6, 2018, 6:00pm UTC](https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971/1 "2018-07-06T18:00:39Z")

</div>

The grok pattern ha an extra escape \ in it in the ingest pip line.

> <https://github.com/elastic/beats/blob/6f99a9627b07aa4bbf00208bcc46d550cb5d563b/filebeat/module/mongodb/log/ingest/pipeline.json#L7>

should look like this.

`%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} %{WORD:mongodb.log.component} *\[%{WORD:mongodb.log.context}\] %{GREEDYDATA:mongodb.log.message}`

Currently using.

ubuntu@ubuntu sudo apt list elasticsearch  
Listing.  
elasticsearch/stable 6.3.1 all [upgradable from: 6.3.0]

ubuntu@ubuntu:~$ sudo apt list filebeat  
Listing.  
filebeat/stable 6.3.1 amd64 [upgradable from: 6.3.0]

---

<div class="post-metadata">

**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\
**Post date:** [July 6, 2018, 8:55pm UTC](https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971/2 "2018-07-06T20:55:44Z")

</div>

With your changes I get

> 2018-07-06T22:52:24.868+0200 ERROR pipeline/output.go:91 Failed to connect: Connection marked as failed because the onConnect callback failed: Error getting pipeline for fileset mongodb/log: Error JSON decoding the pipeline file: ingest/pipeline.json: invalid character '[' in string escape code

The `\\[` escape is used in all the module pipelines, is there because the regular expression is inside a JSON string, so its really read by elasticsearch as `\[`.

Are you experiencing a problem parsing mongodb logs that got fixed by this change?

---

<div class="post-metadata">

**Author:** ![cwray](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@cwray](https://discuss.elastic.co/u/cwray)\
**Post date:** [July 9, 2018, 2:52pm UTC](https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971/3 "2018-07-09T14:52:14Z")

</div>

With the Current grok pattern, i'm getting a grok parse error for all logs coming from mongo 3.4. Beats is currently sending the message. But it is erroring out in the ingest pipeline.

error.message:Provided Grok expressions do not match field value: [2018-07-09T14:49:47.006+0000 I - [conn208899] Index Build (background): 250096500/463852976 53%]

This would probably be more proper for the 3 types of logs that get populated into mongo log.  
`%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} (-|%{WORD:mongodb.log.component}) *\\[%{WORD:mongodb.log.context}\\] %{GREEDYDATA:mongodb.log.message}`

---

<div class="post-metadata">

**Author:** ![cwray](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@cwray](https://discuss.elastic.co/u/cwray)\
**Post date:** [July 10, 2018, 1:47am UTC](https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971/5 "2018-07-10T01:47:50Z")

</div>

Managed to get this all to work by adding one more grok line to the patterns list.

```auto
"processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} %{WORD:mongodb.log.component} *\\[%{WORD:mongodb.log.context}\\] %{GREEDYDATA:mongodb.log.message}",
            "%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} - *\\[%{WORD:mongodb.log.context}\\] %{GREEDYDATA:mongodb.log.message}"
          ],

```

---

<div class="post-metadata">

**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\
**Post date:** [July 10, 2018, 8:43am UTC](https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971/6 "2018-07-10T08:43:04Z")

</div>

Thanks @cwray, now it makes more sense.

Can you open a Pull-Request to the beats repo or prefer if I do it for you?

---

<div class="post-metadata">

**Author:** ![cwray](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@cwray](https://discuss.elastic.co/u/cwray)\
**Post date:** [July 10, 2018, 3:33pm UTC](https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971/7 "2018-07-10T15:33:13Z")

</div>

> <https://github.com/elastic/beats/pull/7560>

Pull-request sent. It is stating that i did not sign the CLA. I went through the signing process before submitting the pull request. So I don't know what is going on there.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 7, 2018, 5:33pm UTC](https://discuss.elastic.co/t/filebeat-mongo-module-grok-pattern-is-incorrect/138971/8 "2018-08-07T17:33:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.