Filebeat conditional output logstash

Elastic Stack Beats
1

Filebeat 6.2

I am using filebeat to pick up two files and send them to two different logstash endpoints.

My config looks like:

---
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - path/to/log1.log 
  multiline.pattern: '^[[:space:]]+\{'                                                                                                                
  multiline.negate: true
  multiline.match: after
  processors:
    - decode_json_fields:
        fields: ["message"]
  tags: ["airdrop"]

- type: log
  enabled: true
  paths:
    - /path/to/log2.log 

processors:
- drop_fields:
    fields: ["beat.name", "beat.hostname", "beat.version", "beat", "host", "input_type", "source", "prospector.type"]

output.logstash:
  when:
   contains:
     tags: tag1
  hosts: ["domain1.com:443"]
  ssl.certificate_authorities: ["/pat/to/cert/cloud-chain-sha512.crt"]
  loadbalance: true

  when:
    contains:
      tags: airdrop
  hosts: ["domain2.com:443"]
  ssl.certificate_authorities: ["path/to/cert/cloud-chain-sha512.crt"]
  loadbalance: true

Contents of my file look like:

   {
      "timestamp" : "2018-04-19 07:34:54.895609-0400",
      "source" : null,
      "messageType" : "Default",
      "processID" : 633,
      "category" : "AirDrop",
      "traceID" : 9203118583709700,
      "machTimestamp" : 32749127946,
      "eventMessage" : "Loading all persisted transfers",
      "activityIdentifier" : 0,
      "subsystem" : "com.apple.sharing",
      "signpostID" : 0,
      "processImageUUID" : "34BD8CEF-7770-3DF6-9B02-B32788EA0C35",
      "threadID" : 3889,
      "senderImageUUID" : "34BD8CEF-7770-3DF6-9B02-B32788EA0C35",
      "parentActivityIdentifier" : 0,
      "eventType" : "logEvent",
      "processImagePath" : "/usr/libexec/sharingd",
      "senderProgramCounter" : 823548,
      "senderImagePath" : "/usr/libexec/sharingd",
      "timezoneName" : ""
   },
   {
      "processImageUUID" : "34BD8CEF-7770-3DF6-9B02-B32788EA0C35",
      "threadID" : 3889,
      "senderImageUUID" : "34BD8CEF-7770-3DF6-9B02-B32788EA0C35",
      "parentActivityIdentifier" : 0,
      "processImagePath" : "/usr/libexec/sharingd",
      "eventType" : "logEvent",
      "timezoneName" : "",
      "senderImagePath" : "/usr/libexec/sharingd",
      "senderProgramCounter" : 1086398,
      "source" : null,
      "timestamp" : "2018-04-19 08:04:51.359642-0400",
      "processID" : 633,
      "category" : "AirDrop",
      "messageType" : "Default",
      "machTimestamp" : 1828470400459,
      "traceID" : 9261530138935300,
      "eventMessage" : "startSending, validated airdrop items. properties: {\n    ConvertMediaFormats = 0;\n    Files =     (\n                {\n            ConvertMediaFormats = 0;\n            FileBomPath = \"./WP-Disaster-Recovery-Apache-Kafka.pdf\";\n            FileIsDirectory = 0;\n            FileName = \"WP-Disaster-Recovery-Apache-Kafka.pdf\";\n            FileType = \"com.adobe.pdf\";\n        }\n    );\n    ReceiverComputerName = dca9049727a4;\n    ReceiverID = b22b03e0f785;\n    VerifiableIdentity = 0;\n}",
      "activityIdentifier" : 0,
      "signpostID" : 0,
      "subsystem" : "com.apple.sharing"
   }
]

My issue is that my multiline confi isn't working. Can someone tell my what it is I am doing wrong with it? I thought I was saying look for something starting with space+{ and create the single line out of that. But I am not getting that at the moment. When I run the config as is I get json parse errors and strange looking fileds.

Any help would be great.

2

To test your multiline config I encourage to have a look at this: https://www.elastic.co/guide/en/beats/filebeat/current/_test_your_regexp_pattern_for_multiline.html Here you can test the regexp with your log line.

3

I fixed the regex by using 'exclude_lines:` . My config now looks like this:


---
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /path/to/file.json
  exclude_lines: ['^\[|^\]'] 
  multiline.pattern: '^[[:space:]]+{'
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: '^[[:space:]]+}|^[[:space:]]+},'                                                                                                                                             

  processors:
    - decode_json_fields:
        fields: ["message"]
  tags: ["airdrop"]

What I think I am doing here is, excluding all single lines that start with ] or [. My Json doc has beginning [ ]. I am then matching a line starting with any number of spaces and { and all negative lines after are appended to the end of the matching {. As an extra precaution I am using a multiline.flush_pattern: '^[[:space:]]+}|^[[:space:]]+},' which I believe is telling the multiline pattern that the last item in the json line will be a } or a },.

My issues is that the last json document in the file gets parsed incorrectly and results in an error.

[2018-04-20T11:51:23,704][ERROR][logstash.codecs.json     ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('=' (code 61)): was expecting a colon to separate field name and value
 at [Source: (String)"{"threadID"=>3889, "senderImageUUID"=>"34BD8CEF-7770-3DF6-9B02-B32788EA0C35", "processImagePath"=>"/usr/libexec/sharingd", "timestamp"=>"2018-04-19 08:04:51.359642-0400", "processImageUUID"=>"34BD8CEF-7770-3DF6-9B02-B32788EA0C35", "parentActivityIdentifier"=>0, "machTimestamp"=>1828470400459, "eventType"=>"logEvent", "processID"=>633, "traceID"=>9261530138935300, "category"=>"AirDrop", "eventMessage"=>"startSending, validated airdrop items. properties: {\n    ConvertMediaFormats = 0;\n    Files "[truncated 621 chars]; line: 1, column: 13]>, :data=>"{\"threadID\"=>3889, \"senderImageUUID\"=>\"34BD8CEF-7770-3DF6-9B02-B32788EA0C35\", \"processImagePath\"=>\"/usr/libexec/sharingd\", \"timestamp\"=>\"2018-04-19 08:04:51.359642-0400\", \"processImageUUID\"=>\"34BD8CEF-7770-3DF6-9B02-B32788EA0C35\", \"parentActivityIdentifier\"=>0, \"machTimestamp\"=>1828470400459, \"eventType\"=>\"logEvent\", \"processID\"=>633, \"traceID\"=>9261530138935300, \"category\"=>\"AirDrop\", \"eventMessage\"=>\"startSending, validated airdrop items. properties: {\\n    ConvertMediaFormats = 0;\\n    Files =     (\\n                {\\n            ConvertMediaFormats = 0;\\n            FileBomPath = \\\"./WP-Disaster-Recovery-Apache-Kafka.pdf\\\";\\n            FileIsDirectory = 0;\\n            FileName = \\\"WP-Disaster-Recovery-Apache-Kafka.pdf\\\";\\n            FileType = \\\"com.adobe.pdf\\\";\\n        }\\n    );\\n    ReceiverComputerName = XXXXXX;\\n    ReceiverID = XXXXXX;\\n    VerifiableIdentity = 0;\\n}\", \"timezoneName\"=>\"\", \"subsystem\"=>\"com.apple.sharing\", \"activityIdentifier\"=>0, \"messageType\"=>\"Default\", \"senderImagePath\"=>\"/usr/libexec/sharingd\", \"source\"=>nil, \"signpostID\"=>0, \"senderProgramCounter\"=>1086398}"}

I have another document in the file that is almost the exact thing but because it is not the last item it does not throw the same error. I have tried ending the file with other documents and it results in the same error.

What am I missing?

4

An important note for the exclude_line is that it gets applied only after the multiline has done. Still I would expect your above code to work except that some events probably contain a , at the end and indentation is not correct (not sure if that matters for LS)

The error you posted above comes from LS. So it seems you also do some json decoding on the LS side? What does your LS config look like?

Can you try to use the file output and see if in this file you get the expected output?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.