Filebeat CEF Module

Elastic Stack Beats
1

We've been trying to create a pipeline for logs from a security tool using the recently released CEF module, but we've been getting an error about the log format and its parsing.

The tool would pull its logs via an API call and then it will send it over syslog to a localhost. Filebeat with the CEF module would run on the same host listing on the syslog port.

So, we're seeing this.

$sudo lsof -i :514

vendor_agent 1132 daemon 8u  IPv4  17481 0t0  UDP localhost:53376->localhost:syslog 
filebeat  2281    root   6u  IPv4  21113 0t0  UDP localhost:syslog

Next, we looked into the the log format being generated by the vendor tool, and it looks like this, good we want CEF formatted logs.

"CEF:0|Vendor|1.0|xxx|xxx|1|cat=AuthActivityAuditEvent destinationTranslatedAddress=xxxx duser=xxxx deviceProcessName=xxx Authentication cn3Label=Offset cn3=11105 outcome=true\n"

Finally, reading this document: CEF module | Filebeat Reference [8.11] | Elastic, and just looking into the first paragraph, this should work.

....This is a module for receiving Common Event Format (CEF) data over Syslog. When messages are received over the syslog protocol the syslog input will parse the header and set the timestamp value. Then the decode_cef processor is applied to parse the CEF encoded data. The decoded data is written into a cef object field. Lastly any Elastic Common Schema (ECS) fields that can be populated with the CEF data are populated.

But, we're getting an error for the above pipeline.

ERROR [syslog] syslog/input.go:134 can't parse event as syslog rfc3164 {"message": xxxxxxxx}

2

AFAICS, the line format doesn't follow the expected format from the CEF module. @adrisr maybe you can help @daniel_a with this?

3

The syslog input is failing to parse the syslog header. We're seeing this problem a lot because Filebeat's syslog input is too strict and only supports BSD-style RFC3164 messages.

In your case it might be related to the date format that your CEF exporter is using. Do you have a config option to change it?

As an alternative, you can modify the module to use the udp input instead of the syslog input, which does no parsing. See this message:

The file you need to change is module/cef/log/config/input.yml under /usr/share/filebeat/....

1 Like
4

Adjusting /usr/share/filebeat/module/cef/log/config/input.yml seems to fix the issue with the pipeline. Right now, I can see the logs are being ingested but they are not being parsed. The vendor log gets send to a message field.

@timestamp	Nov 4, 2019 @ 15:06:32.462
_id	 <number>
_index	vendor-2019.11.04
_score	 - 
_type	_doc
agent.ephemeral_id	67859a-1618-4242
agent.hostname	<name>
agent.id	4159-4ccd-baa6
agent.type	filebeat
agent.version	7.4.1
ecs.version	1.1.0
log.file.path	/var/log/output
log.offset	1,068
message	CEF:0|Vendor|Tool|1.0|saml2Assert|saml2Assert|1|cat=AuthActivityAuditEvent 
destinationTranslatedAddress=IP duser=name deviceProcessName=Vendor Authentication 
cn3Label=Offset cn3=11158 outcome=true deviceCustomDate1Label=Timestamp 
deviceCustomDate1=Nov 04 2019 23:06:24 rt=3408309840
5

@daniel_a was this log line parsed by the CEF module? It looks like it comes from a log file source instead of UDP/syslog.

6

I just noticed the log.file.path line above in the post. It's weird since I have disabled type: log in the configuration file.

filebeat.inputs:
  - type: log
    enabled: false
    paths:
      - /var/log/output
7

This what I'm getting (in Kinbana) after disabling the option to write logs to an output file in the vendor agent configuration file, not in filbebeat configuration file. Still not sure why the filebeat configuration enabled: false switch is not working.

cef.device.event_class_id	saml2Assert
cef.device.product	Vendor
cef.device.vendor	Vendor
cef.device.version	1.0
cef.extensions.destinationTranslatedAddress	x.x.x.x
cef.extensions.destinationUserName	<emial address>
cef.extensions.deviceCustomDate1	Nov 05 2019 18:46:03
cef.extensions.deviceCustomDate1Label	Timestamp
cef.extensions.deviceCustomNumber3	11180
cef.extensions.deviceCustomNumber3Label	Offset
cef.extensions.deviceEventCategory	AuthActivityAuditEvent
cef.extensions.deviceProcessName  Vendor Authentication
cef.extensions.deviceReceiptTime	1572979563152
cef.extensions.eventOutcome	true
cef.name	saml2Assert
cef.severity	1
cef.version	0

@adrisr is this ^ the desired output from the CEF module, is this the ECS format? I can't find anything similar under here: https://www.elastic.co/guide/en/ecs/current/index.html.

I'm also seeing the following error:

deviceReceiptTime: value is not a valid timestamp

--Thanks

8

That's part of the output of the CEF module. You should also see some ECS fields populated, in this case:

  • destination.nat.ip: x.x.x.x
  • destination.user.name: <email.address>
  • process.name: "Vendor Authentication"

About the error, can you paste the (redacted) event that contains this error?

1 Like
9

This makes sense now. I can see two fields: a field with a cef prefix and the one which was translated from it to the ECS format.

The error is related to the epoch time format, look below. I can fix this.

cef.extensions.deviceReceiptTime 1572986753641 text

Thank you again for your quick responses.

--Daniel

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.