Eventstore Grok or any other compatible filter

Elastic Stack Logstash
1

I am stuck trying to find a compatible eventstore parser that can filter out error logs coming from an eventstore server through filebeat, just like APACHELOG, SYSLOG and others have their own.
If there's no specific filter for eventstore, what would be the best way to parse it?

Example of a message that I'd like to parse is:

{
  "_index": "dev-logs-eventstore-bettingengine-2019.09.16",
  "_type": "log",
  "_id": "AW07MzveiJ_5Y3xfUSd7",
  "_version": 1,
  "_score": null,
  "_source": {
    "server_name": "dev-events01",
    "count": 1,
    "source": "/var/log/eventstore/whatever/eventstore-bettingengine40.log",
    "message": "{ \"PID\": \"115348\", \"ThreadID\": \"13\", \"Date\": \"2019-09-16T18:30:19.221868Z\", \"Level\": \"Trace\", \"Logger\": \"GossipServiceBase\", \"Message\": \"CLUSTER HAS CHANGED {source}\\nOld:\\n{@oldMembers}\\nNew:\\n{@newMembers}\", \"EventProperties\": { \"source\": \"gossip received from [10.15.126.12:2112]\", \"oldMembers\": [\"VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n\\/a, 10.15.126.13:1113, n\\/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216\\/5256872099\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352\",\"VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [Slave, 10.15.126.12:1112, n\\/a, 10.15.126.12:1113, n\\/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.349\",\"VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352\"], \"newMembers\": [\"VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n\\/a, 10.15.126.13:1113, n\\/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216\\/5256872099\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221\",\"VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [PreReplica, 10.15.126.12:1112, n\\/a, 10.15.126.12:1113, n\\/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.217\",\"VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221\"] } }",
    "tags": [
      "beats_input_codec_plain_applied",
      "filtered_by_grok",
      "_dateparsefailure"
    ],
    "@timestamp": "2019-09-16T17:51:12.607Z",
    "@version": "1",
    "beat": {
      "name": "dev-events01",
      "hostname": "dev-events01"
    },
    "timestamp": "2019-09-16T18:30:19.221868Z"
  },
  "fields": {
    "@timestamp": [
      1568656272607
    ]
  },
  "sort": [
    1568656272607
  ]
}
2

@Ghassan_Zein

Could you provide what you want the result to look like?

It seems to me that the message field is a valid JSON object. So, it should first be passed through the JSON processor.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "json": {
          "field": "message"
        }
      }
    ]
  },
  "docs": [
    {
  "_index": "dev-logs-eventstore-bettingengine-2019.09.16",
  "_id": "AW07MzveiJ_5Y3xfUSd7",
  "_source": {
    "server_name": "dev-events01",
    "count": 1,
    "source": "/var/log/eventstore/whatever/eventstore-bettingengine40.log",
    "message": "{ \"PID\": \"115348\", \"ThreadID\": \"13\", \"Date\": \"2019-09-16T18:30:19.221868Z\", \"Level\": \"Trace\", \"Logger\": \"GossipServiceBase\", \"Message\": \"CLUSTER HAS CHANGED {source}\\nOld:\\n{@oldMembers}\\nNew:\\n{@newMembers}\", \"EventProperties\": { \"source\": \"gossip received from [10.15.126.12:2112]\", \"oldMembers\": [\"VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n\\/a, 10.15.126.13:1113, n\\/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216\\/5256872099\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352\",\"VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [Slave, 10.15.126.12:1112, n\\/a, 10.15.126.12:1113, n\\/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.349\",\"VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352\"], \"newMembers\": [\"VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n\\/a, 10.15.126.13:1113, n\\/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216\\/5256872099\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221\",\"VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [PreReplica, 10.15.126.12:1112, n\\/a, 10.15.126.12:1113, n\\/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.217\",\"VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216\\/5256868822\\/5256868822\\/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221\"] } }",
    "tags": [
      "beats_input_codec_plain_applied",
      "filtered_by_grok",
      "_dateparsefailure"
    ],
    "@timestamp": "2019-09-16T17:51:12.607Z",
    "@version": "1",
    "beat": {
      "name": "dev-events01",
      "hostname": "dev-events01"
    },
    "timestamp": "2019-09-16T18:30:19.221868Z"
  }
}]
}

Results in

{
  "docs" : [
    {
      "doc" : {
        "_index" : "dev-logs-eventstore-bettingengine-2019.09.16",
        "_type" : "_doc",
        "_id" : "AW07MzveiJ_5Y3xfUSd7",
        "_source" : {
          "server_name" : "dev-events01",
          "@timestamp" : "2019-09-16T17:51:12.607Z",
          "count" : 1,
          "@version" : "1",
          "beat" : {
            "name" : "dev-events01",
            "hostname" : "dev-events01"
          },
          "source" : "/var/log/eventstore/whatever/eventstore-bettingengine40.log",
          "message" : {
            "Message" : """
CLUSTER HAS CHANGED {source}
Old:
{@oldMembers}
New:
{@newMembers}
""",
            "ThreadID" : "13",
            "EventProperties" : {
              "newMembers" : [
                "VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n/a, 10.15.126.13:1113, n/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216/5256872099/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221",
                "VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [PreReplica, 10.15.126.12:1112, n/a, 10.15.126.12:1113, n/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216/5256868822/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.217",
                "VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216/5256868822/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:19.221"
              ],
              "oldMembers" : [
                "VND {fe924279-395f-4272-888e-b24a447c0093} <LIVE> [Master, 10.15.126.13:1112, n/a, 10.15.126.13:1113, n/a, 10.15.126.13:2112, 10.15.126.13:2113] 5256863216/5256872099/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352",
                "VND {4c86ac49-d9ed-46d3-8be1-f8b83ce880ff} <LIVE> [Slave, 10.15.126.12:1112, n/a, 10.15.126.12:1113, n/a, 10.15.126.12:2112, 10.15.126.12:2113] 5256863216/5256868822/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.349",
                "VND {1d336642-e9b2-49bd-97e4-479c3e26dd9f} <LIVE> [PreReplica, 10.15.126.11:1112, 10.15.126.11:0, 10.15.126.11:1113, 10.15.126.11:0, 10.15.126.11:2112, 10.15.126.11:2113] 5256863216/5256868822/5256868822/E5@4803775845:{bcc8060c-433d-47c0-9889-64875bb7aa06} | 2019-09-11 01:20:18.352"
              ],
              "source" : "gossip received from [10.15.126.12:2112]"
            },
            "PID" : "115348",
            "Level" : "Trace",
            "Logger" : "GossipServiceBase",
            "Date" : "2019-09-16T18:30:19.221868Z"
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "filtered_by_grok",
            "_dateparsefailure"
          ],
          "timestamp" : "2019-09-16T18:30:19.221868Z"
        },
        "_ingest" : {
          "timestamp" : "2019-09-18T11:12:35.122705Z"
        }
      }
    }
  ]
}
3

Thank you for coming back BenTrent.

What am trying to do is to create a filter on logstash that filters out eventstore's log for this case.

How should I configure this JSON processor to run as filter in logstash for the whole log file I sent?

The way you sent Results is acceptable, just need to integrate it. Please advise and thanks again.

4

@Ghassan_Zein

You can either use a defined pipeline on an Elasticsearch Ingest node https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline.html and refer to it via logstash https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html

Or use the JSON filter in logstash https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

The choice is yours.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.