Filebeat character encoding problem!

Elastic Stack Beats
1

It looks like filebeat didn't handle > character properly and displayed the Unicode hex equivalent i.e. \u003c142 in output. Other fields are displayed just fine.

Any recommendations?

Filebeat version: 6.3

Input: SYSLOG or TCP [Both input types are having this problem]
Destination: Kafka

Error log:

"host": {
    "name": "COMPUTERNAME"
  }
}
2018-06-19T09:13:38.770Z	ERROR	[syslog]	syslog/input.go:114	can't not parse event as syslog rfc3164	{"message": "<190>2018-06-19 02:13:38 Local7.Info HOST-1111-PQWER-SWT1 2018-06-19T02:13:38.635322-07:00 HOST-1111-PQWER-SWT1 Acl: %ACL-6-IPACCESS: list Isolation-4E Vlan100 denied tcp 10.10.10.10(56393) -> 13.14.14.53(443)"}
2018-06-19T09:13:38.775Z	DEBUG	[publish]	pipeline/processor.go:291	Publish event: {
  "@timestamp": "2018-06-19T09:13:38.770Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.0",
    "truncated": false
  },
  "prospector": {
    "type": "syslog"
  },
  "host": {
    "name": "COMPUTERNAME"
  },
  "beat": {
    "name": "COMPUTERNAME",
    "hostname": "COMPUTERNAME",
    "version": "6.3.0"
  },
  "message": "\u003c190\u003e2018-06-19 02:13:38 Local7.Info HOST-1111-PQWER-SWT1 2018-06-19T02:13:38.635322-07:00 HOST-1111-PQWER-SWT1 Acl: %ACL-6-IPACCESS: list Isolation-4E Vlan100 denied tcp 10.10.10.10(56393) -**\u003e** 13.14.14.53(443)
2

Please share the Filebeat configuration that you are using.

4 
filebeat.inputs:
- type: tcp
  host: "localhost:516"
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  # Set to true to enable config reloading
  reload.enabled: false
output.kafka:
 enabled: true
 #configure topic as per your application need
 hosts: ["kafkabroker1:9092", "kafkabroker2:9092", "kafkabroker3:9092", "kafkabroker4:9092",
"kafkabroker5:9092",
"kafkabroker6:9092",
"kafkabroker7:9092",
"kafkabroker8:9092",
"kafkabroker9:9092",
"kafkabroker10:9092"]
 topic: syslog_cisco
logging.level: debug
logging.to_files: true
logging.files:
 path: C:\Filebeat\Logs
 name: filebeat.log
 rotateeverybytes: 10485760
 keepfiles: 7
5

Filebeat is receiving your data fine, but when it encodes the message to JSON it escapes the HTML characters. See https://github.com/elastic/beats/issues/2581.

Regarding the syslog parser error, that seems to be caused by the format of the date. 2018-06-19 02:13:38 gets rejected but something like 2018-06-19T02:13:38 would be fine. I'm not a RFC3164 expert, but from looking at the parser and spec that seems to be the cause of the parse failure.

6

That's a problem for us. Syslog event displays event priority inside <>. Is there any way I can disable HTML escaping when marshaling JSON with filebeat?

Also, Is it possible to define datetime format inside filebeat config such as %Y-%m-%d %H:%M:%S

7

There isn't yet. Please comment on github issue to help move the discussion along.

The syslog parser is not configurable. But perhaps it can be enhanced to handle more date formats. My only worry is that it then might not be RFC compliant. What does @pierhugues think?

8

Currently for simplicity and speed the parser doesn't allow to configure the date pattern.
But this data format should be parsed.

RFC compliance can be a bit loose for syslog.

I will add that format to the parser, this should not be much trouble.

9

@Pratik.Lal What device is emitting this log and can you check if the following mapping would work?

The original string

<190>2018-06-19 02:13:38 Local7.Info HOST-1111-PQWER-SWT1 2018-06-19T02:13:38.635322-07:00 HOST-1111-PQWER-SWT1 Acl: %ACL-6-IPACCESS: list Isolation-4E Vlan100 denied tcp 10.10.10.10(56393)

I think the following would be the correct mapping.

190 is the priority,
2018-06-19 02:13:38 should be the date.
Local7.Info hostname
HOST-1111-PQWER-SWT1 The software or appliance.
2018-06-19T02:13:38.635322-07:00 HOST-1111-PQWER-SWT1 Acl: %ACL-6-IPACCESS: list Isolation-4E Vlan100 denied tcp 10.10.10.10(56393) should be the message

The message parts contain another date, this is not part of the original Syslog RFC, but it could be extracted with a custom ingest pipeline.

The current date format trips the parser but adding support for it should not be too hard.

10

@Pratik.Lal is there is a relay host in the equation?

11

@pierhugues Yep... we're receving logs from a centralized syslog server and not from actual appliance (CISCO NX-OS). Also, mapping suggested by you should work.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.