Filebeat 7.3.0 froze parsing multiline json

Elastic Stack Beats
1

Hi Team,

I started with ELK 7.3.0 for SIEM feature. I now need to integrate Cloudflare logs which is in JSON format. I'm pretty sure that beats can do that.
I have installed filebeat rpm package

What I have done so far...

Elastic port changed to : 1300
Logstash port changed to: 2300
Kibana port changed to: 3300

Setup filebeat.yml

filebeat.config.inputs:
  enabled: true
  path: configs/*.yml
  reload.enabled: true
  reload.period: 10s

setup.kibana: 
  host: "10.139.111.216:3300"

output.elasticsearch:
  hosts: ["10.139.111.216:1300"]

I have placed my config in /etc/filebeat/modules.d/cloudflare-FW.conf directory as this will reload if any config changes made as I have mentioned in filebeat.yml

cloudflare-FW.conf

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - /var/crons/logs/*.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

processors:
- decode_json_fields:
 fields: ['message']
 target: json

output.elasticsearch:
  hosts: ["10.136.111.216:1300"]
  index: "cloudflare-FW-%{+yyyy.MM.dd}"
  setup.template.enabled: true
  setup.template.name: "cloudflare-FW"
  setup.template.pattern: "cloudflare-FW-*"

This is my target json file to parse

  {
      "ray_id": "501dxxxxxxxx0ce1b",
      "kind": "firewall",
      "source": "bic",
      "action": "drop",
      "rule_id": "bic",
      "ip": "161.00.6.8",
      "ip_class": "noRecord",
      "country": "GB",
      "colo": "LHR",
      "host": "site.com",
      "method": "POST",
      "proto": "HTTP/1.1",
      "scheme": "https",
      "ua": "Jakarta Commons-HttpClient/3.1",
      "uri": "/nxxxxxxx/senxxxxxxxxail.php",
      "matches": [
        {
          "rule_id": "bic",
          "source": "bic",
          "action": "drop"
        }
      ],
      "occurred_at": "2019-08-06T03:28:57Z"
    },
    {
      "ray_id": "501dxxxxxxxxxx6ce27",
      "kind": "firewall",
      "source": "bic",
      "action": "drop",
      "rule_id": "bic",
      "ip": "161.61.6.8",
      "ip_class": "noRecord",
      "country": "GB",
      "colo": "LHR",
      "host": "site.com",
      "method": "POST",
      "proto": "HTTP/1.1",
      "scheme": "https",
      "ua": "Jakarta Commons-HttpClient/3.1",
      "uri": "/nxxxxxxx/senxxxxxxxxail.php",
      "matches": [
        {
          "rule_id": "bic",
          "source": "bic",
          "action": "drop"
        }
      ],
      "occurred_at": "2019-08-06T03:02:52Z"
    }

I expected when running filebeat -e an index will get automatically created as specified and logs will get pushed to the index. But it is not happening so.

Any assistance is greatly appreciated

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.