Unable to parse JSON logs through filebeat prospector

Elastic Stack Beats
1

I'm trying to parse multiline JSON logs into elasticsearch using filebeat.

{
  "took": 95,
  "timed_out": false,
  "_shards": {
    "total": 85,
    "successful": 85,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 391847,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "metrics": {
      "doc_count_error_upper_bound": 15546,
      "sum_other_doc_count": 365359,
      "buckets": [
        {
          "key": "ilb.io",
          "doc_count": 26488,
          "network_name": {
            "doc_count_error_upper_bound": 1211,
            "sum_other_doc_count": 25508,
            "buckets": [
              {
                "key": "enp130s17f2",
                "doc_count": 980,
                "series": {
                  "buckets": [
                    {
                      "key_as_string": "2017-12-15T04:00:00.000Z",
                      "key": 1513310400000,
                      "doc_count": 231,
                      "avg": {
                        "value": 0
                      }
                    },
                    {
                      "key_as_string": "2017-12-15T08:00:00.000Z",
                      "key": 1513324800000,
                      "doc_count": 749,
                      "avg": {
                        "value": 0
                      },
                      "movavg": {
                        "value": 0
                      },
                      "surprise": {
                        "value": 0
                      }
                    }
                  ]
                },
                "largest_surprise": {
                  "value": 0,
                  "keys": [
                    "2017-12-15T08:00:00.000Z"
                  ]
                }
              }
            ]
          },
          "ninetieth_surprise": {
            "values": {
              "90.0": 0
            }
          }
        }
      ]
    }
  }
}

And the filebeat.yml config:

  json.message_key: aggregations
  json.keys_under_root: true
  json.add_error_key: true
  tags: [
    "aggs",
    "anomaly",
    "json"
  ]

could somebody please point me in right direction,..??

2

Hi @Shubham_Mahajan,

When parsing a multiline JSON message first you have to get the full message using multiline and then apply decode_json_fileds processor to the result, try something like this:

filebeat.prospectors:
- paths:
    - input.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after
  processors:
  - decode_json_fields:
      fields: ['message']
      target: json
3

Thanks a lot @exekias

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.