Filebeat 7.3.0 froze parsing multiline json

Hi Team,

I started with ELK 7.3.0 for SIEM feature. I now need to integrate Cloudflare logs which is in JSON format. I'm pretty sure that beats can do that.
I have installed filebeat rpm package

What I have done so far...

Elastic port changed to : 1300
Logstash port changed to: 2300
Kibana port changed to: 3300

Setup filebeat.yml

filebeat.config.inputs:
  enabled: true
  path: configs/*.yml
  reload.enabled: true
  reload.period: 10s

setup.kibana: 
  host: "10.139.111.216:3300"

output.elasticsearch:
  hosts: ["10.139.111.216:1300"]

I have placed my config in /etc/filebeat/modules.d/cloudflare-FW.conf directory as this will reload if any config changes made as I have mentioned in filebeat.yml

cloudflare-FW.conf

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - /var/crons/logs/*.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

processors:
- decode_json_fields:
 fields: ['message']
 target: json

output.elasticsearch:
  hosts: ["10.136.111.216:1300"]
  index: "cloudflare-FW-%{+yyyy.MM.dd}"
  setup.template.enabled: true
  setup.template.name: "cloudflare-FW"
  setup.template.pattern: "cloudflare-FW-*"

This is my target json file to parse

  {
      "ray_id": "501dxxxxxxxx0ce1b",
      "kind": "firewall",
      "source": "bic",
      "action": "drop",
      "rule_id": "bic",
      "ip": "161.00.6.8",
      "ip_class": "noRecord",
      "country": "GB",
      "colo": "LHR",
      "host": "site.com",
      "method": "POST",
      "proto": "HTTP/1.1",
      "scheme": "https",
      "ua": "Jakarta Commons-HttpClient/3.1",
      "uri": "/nxxxxxxx/senxxxxxxxxail.php",
      "matches": [
        {
          "rule_id": "bic",
          "source": "bic",
          "action": "drop"
        }
      ],
      "occurred_at": "2019-08-06T03:28:57Z"
    },
    {
      "ray_id": "501dxxxxxxxxxx6ce27",
      "kind": "firewall",
      "source": "bic",
      "action": "drop",
      "rule_id": "bic",
      "ip": "161.61.6.8",
      "ip_class": "noRecord",
      "country": "GB",
      "colo": "LHR",
      "host": "site.com",
      "method": "POST",
      "proto": "HTTP/1.1",
      "scheme": "https",
      "ua": "Jakarta Commons-HttpClient/3.1",
      "uri": "/nxxxxxxx/senxxxxxxxxail.php",
      "matches": [
        {
          "rule_id": "bic",
          "source": "bic",
          "action": "drop"
        }
      ],
      "occurred_at": "2019-08-06T03:02:52Z"
    }

I expected when running filebeat -e an index will get automatically created as specified and logs will get pushed to the index. But it is not happening so.

Any assistance is greatly appreciated

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.