# Eventstore Grok or any other compatible filter

**URL:** <https://discuss.elastic.co/t/eventstore-grok-or-any-other-compatible-filter/199768>\
**Category:** Logstash\
**Created:** [September 17, 2019, 7:23am UTC](https://discuss.elastic.co/t/eventstore-grok-or-any-other-compatible-filter/199768 "2019-09-17T07:23:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ghassan\_Zein](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghassan_zein/32/54311_2.png) [@Ghassan\_Zein](https://discuss.elastic.co/u/Ghassan_Zein)\
**Post date:** [September 17, 2019, 7:23am UTC](https://discuss.elastic.co/t/eventstore-grok-or-any-other-compatible-filter/199768/1 "2019-09-17T07:23:42Z")

</div>

I am stuck trying to find a compatible eventstore parser that can filter out error logs coming from an eventstore server through filebeat, just like APACHELOG, SYSLOG and others have their own.  
If there's no specific filter for eventstore, what would be the best way to parse it?

Example of a message that I'd like to parse is:

```
{
  "_index": "dev-logs-eventstore-bettingengine-2019.09.16",
  "_type": "log",
  "_id": "AW07MzveiJ_5Y3xfUSd7",
  "_version": 1,
  "_score": null,
  "_source": {
    "server_name": "dev-events01",
    "count": 1,
    "source": "/var/log/eventstore/whatever/eventstore-bettingengine40.log",
    "message": "{ \"PID\": \"115348\", \"ThreadID\": \"13\", \"Date\": \"2019-09-16T18:30:19.221868Z\", \"Level\": \"Trace\", \"Logger\": \"GossipServiceBase\", \"Message\": \"CLUSTER HAS CHANGED {source}\\nOld:\\n{@oldMembers}\\nNew:\\n{@newMembers}\", \"EventProperties\": { \"source\": \"gossip received from [10.15.126.12:2112]\", \"oldMembers\": [\"VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n\\/a, 10.15.126.13:1113, n\\/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216\\/5256872099\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352\",\"VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [Slave, 10.15.126.12:1112, n\\/a, 10.15.126.12:1113, n\\/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.349\",\"VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352\"], \"newMembers\": [\"VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n\\/a, 10.15.126.13:1113, n\\/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216\\/5256872099\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221\",\"VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [PreReplica, 10.15.126.12:1112, n\\/a, 10.15.126.12:1113, n\\/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.217\",\"VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221\"] } }",
    "tags": [
      "beats_input_codec_plain_applied",
      "filtered_by_grok",
      "_dateparsefailure"
    ],
    "@timestamp": "2019-09-16T17:51:12.607Z",
    "@version": "1",
    "beat": {
      "name": "dev-events01",
      "hostname": "dev-events01"
    },
    "timestamp": "2019-09-16T18:30:19.221868Z"
  },
  "fields": {
    "@timestamp": [
      1568656272607
    ]
  },
  "sort": [
    1568656272607
  ]
}
```

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [September 18, 2019, 11:16am UTC](https://discuss.elastic.co/t/eventstore-grok-or-any-other-compatible-filter/199768/2 "2019-09-18T11:16:05Z")

</div>

@Ghassan_Zein

Could you provide what you want the result to look like?

It seems to me that the `message` field is a valid JSON object. So, it should first be passed through the [JSON processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/json-processor.html).

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "json": {
          "field": "message"
        }
      }
    ]
  },
  "docs": [
    {
  "_index": "dev-logs-eventstore-bettingengine-2019.09.16",
  "_id": "AW07MzveiJ_5Y3xfUSd7",
  "_source": {
    "server_name": "dev-events01",
    "count": 1,
    "source": "/var/log/eventstore/whatever/eventstore-bettingengine40.log",
    "message": "{ \"PID\": \"115348\", \"ThreadID\": \"13\", \"Date\": \"2019-09-16T18:30:19.221868Z\", \"Level\": \"Trace\", \"Logger\": \"GossipServiceBase\", \"Message\": \"CLUSTER HAS CHANGED {source}\\nOld:\\n{@oldMembers}\\nNew:\\n{@newMembers}\", \"EventProperties\": { \"source\": \"gossip received from [10.15.126.12:2112]\", \"oldMembers\": [\"VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n\\/a, 10.15.126.13:1113, n\\/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216\\/5256872099\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352\",\"VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [Slave, 10.15.126.12:1112, n\\/a, 10.15.126.12:1113, n\\/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.349\",\"VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352\"], \"newMembers\": [\"VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n\\/a, 10.15.126.13:1113, n\\/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216\\/5256872099\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221\",\"VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [PreReplica, 10.15.126.12:1112, n\\/a, 10.15.126.12:1113, n\\/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.217\",\"VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221\"] } }",
    "tags": [
      "beats_input_codec_plain_applied",
      "filtered_by_grok",
      "_dateparsefailure"
    ],
    "@timestamp": "2019-09-16T17:51:12.607Z",
    "@version": "1",
    "beat": {
      "name": "dev-events01",
      "hostname": "dev-events01"
    },
    "timestamp": "2019-09-16T18:30:19.221868Z"
  }
}]
}

```

Results in

```auto
{
  "docs" : [
    {
      "doc" : {
        "_index" : "dev-logs-eventstore-bettingengine-2019.09.16",
        "_type" : "_doc",
        "_id" : "AW07MzveiJ_5Y3xfUSd7",
        "_source" : {
          "server_name" : "dev-events01",
          "@timestamp" : "2019-09-16T17:51:12.607Z",
          "count" : 1,
          "@version" : "1",
          "beat" : {
            "name" : "dev-events01",
            "hostname" : "dev-events01"
          },
          "source" : "/var/log/eventstore/whatever/eventstore-bettingengine40.log",
          "message" : {
            "Message" : """
CLUSTER HAS CHANGED {source}
Old:
{@oldMembers}
New:
{@newMembers}
""",
            "ThreadID" : "13",
            "EventProperties" : {
              "newMembers" : [
                "VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n/a, 10.15.126.13:1113, n/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216/5256872099/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221",
                "VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [PreReplica, 10.15.126.12:1112, n/a, 10.15.126.12:1113, n/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216/5256868822/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.217",
                "VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216/5256868822/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221"
              ],
              "oldMembers" : [
                "VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n/a, 10.15.126.13:1113, n/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216/5256872099/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352",
                "VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [Slave, 10.15.126.12:1112, n/a, 10.15.126.12:1113, n/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216/5256868822/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.349",
                "VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216/5256868822/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352"
              ],
              "source" : "gossip received from [10.15.126.12:2112]"
            },
            "PID" : "115348",
            "Level" : "Trace",
            "Logger" : "GossipServiceBase",
            "Date" : "2019-09-16T18:30:19.221868Z"
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "filtered_by_grok",
            "_dateparsefailure"
          ],
          "timestamp" : "2019-09-16T18:30:19.221868Z"
        },
        "_ingest" : {
          "timestamp" : "2019-09-18T11:12:35.122705Z"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Ghassan\_Zein](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghassan_zein/32/54311_2.png) [@Ghassan\_Zein](https://discuss.elastic.co/u/Ghassan_Zein)\
**Post date:** [September 18, 2019, 12:05pm UTC](https://discuss.elastic.co/t/eventstore-grok-or-any-other-compatible-filter/199768/3 "2019-09-18T12:05:42Z")

</div>

Thank you for coming back BenTrent.

What am trying to do is to create a filter on logstash that filters out eventstore's log for this case.

How should I configure this JSON processor to run as filter in logstash for the whole log file I sent?

The way you sent Results is acceptable, just need to integrate it. Please advise and thanks again.

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [September 18, 2019, 6:34pm UTC](https://discuss.elastic.co/t/eventstore-grok-or-any-other-compatible-filter/199768/4 "2019-09-18T18:34:26Z")

</div>

@Ghassan_Zein

You can either use a defined pipeline on an Elasticsearch Ingest node [https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline.html) and refer to it via logstash [https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html](https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html)

Or use the JSON filter in logstash [https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html)

The choice is yours.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 16, 2019, 6:34pm UTC](https://discuss.elastic.co/t/eventstore-grok-or-any-other-compatible-filter/199768/5 "2019-10-16T18:34:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.