Hello,
We are having problems with the'syslog' input of filebeat. I'm using the script for sending a single log to the filebeat syslog input.
I've noticed that the same message is being parsed because I can see the event on my logger server, and I've also noticed errors being sent to the log file, resulting in the loss of many logs.
Single Log:
Jan 24 2024 11:22:41 : %FTD-6-430003: EventPriority: Low, DeviceUUID: 70278058-1b77-11ec-a090-feeeb3702f49, InstanceID: 4, FirstPacketSecond: 2024-01-24T11:21:03Z, ConnectionID: 40428, AccessControlRuleAction: Allow, SrcIP: 0.0.0.0, DstIP: 92.204.128.155, SrcPort: 62416, DstPort: 4433, Protocol: tcp, IngressInterface: inside, EgressInterface: outside, IngressZone: inside, EgressZone: outside, IngressVRF: Global, EgressVRF: Global, ACPolicy: Pol_Access_Control, AccessControlRuleName: Allow_URLs, Prefilter Policy: Default Prefilter Policy_1, Client: SSL client, ApplicationProtocol: HTTPS, ConnectionDuration: 98, InitiatorPackets: 14, ResponderPackets: 16, InitiatorBytes: 1802, ResponderBytes: 6133, NAPPolicy: Balanced Security and Connectivity, SSLPolicy: None, SSLFlowStatus: Success, SSLCipherSuite: Unknown, SSLCertificate: d08c844c3b312370e87118a194ae6f8d52d3626b, SSLVersion: Unknown, SSLServerCertStatus: Valid, SSLActualAction: Do Not Decrypt, SSLExpectedAction: Do Not Decrypt, SSLServerName: ns1002325.ip-92-204-128.us, URLCategory: Infrastructure and Content Delivery Networks, URLReputation: Neutral, URL: https://ns1002325.ip-92-204-128.us
Filebeat.yml configuration file:
fields_under_root: true
fields.collector_node_id: AgentServer
output.logstash:
hosts: ["xx.xx.xx.xx:xxxx"]
max_procs: 4
queue:
disk:
path: C:\Program Files\sequretek\Syslog\data\diskqueue
max_size: 10GB
segment_size: 1MB
retry_interval: 1s
filebeat.inputs:
- type: syslog
enabled: true
keep_null: true
format: auto
timeout: 10
protocol.udp:
host: "0.0.0.0:514"
- type: syslog
enabled: true
format: auto
timeout: 10
keep_null: true
protocol.tcp:
host: "0.0.0.0:514"
Below is the error getting into log file.
{"log.level":"error","@timestamp":"2024-02-01T19:42:11.413+0530","log.logger":"syslog","log.origin":{"file.name":"syslog/input.go","file.line":285},"message":"can't parse event as syslog rfc3164","service.name":"filebeat","message":"Jan 24 2024 11:22:41 : %FTD-6-430003: EventPriority: Low, DeviceUUID: 70278058-1b77-11ec-a090-feeeb3702f49, InstanceID: 4, FirstPacketSecond: 2024-01-24T11:21:03Z, ConnectionID: 40428, AccessControlRuleAction: Allow, SrcIP: 0.0.0.0, DstIP: 92.204.128.155, SrcPort: 62416, DstPort: 4433, Protocol: tcp, IngressInterface: inside, EgressInterface: outside, IngressZone: inside, EgressZone: outside, IngressVRF: Global, EgressVRF: Global, ACPolicy: Pol_Access_Control, AccessControlRuleName: Allow_URLs, Prefilter Policy: Default Prefilter Policy_1, Client: SSL client, ApplicationProtocol: HTTPS, ConnectionDuration: 98, InitiatorPackets: 14, ResponderPackets: 16, InitiatorBytes: 1802, ResponderBytes: 6133, NAPPolicy: Balanced Security and Connectivity, SSLPolicy: None, SSLFlowStatus: Success, SSLCipherSuite: Unknown, SSLCertificate: d08c844c3b312370e87118a194ae6f8d52d3626b, SSLVersion: Unknown, SSLServerCertStatus: Valid, SSLActualAction: Do Not Decrypt, SSLExpectedAction: Do Not Decrypt, SSLServerName: ns1002325.ip-92-204-128.us, URLCategory: Infrastructure and Content Delivery Networks, URLReputation: Neutral, URL: https://ns1002325.ip-92-204-128.us","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-02-01T19:42:12.398+0530","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(async(tcp://172.24.2.55:7777)): dial tcp 172.24.2.55:7777: connectex: No connection could be made because the target machine actively refused it.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-02-01T19:42:12.398+0530","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Attempting to reconnect to backoff(async(tcp://172.24.2.55:7777)) with 1 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-02-01T19:42:12.415+0530","log.logger":"syslog","log.origin":{"file.name":"syslog/input.go","file.line":285},"message":"can't parse event as syslog rfc3164","service.name":"filebeat","message":"Jan 24 2024 11:22:41 : %FTD-6-430003: EventPriority: Low, DeviceUUID: 70278058-1b77-11ec-a090-feeeb3702f49, InstanceID: 4, FirstPacketSecond: 2024-01-24T11:21:03Z, ConnectionID: 40428, AccessControlRuleAction: Allow, SrcIP: 0.0.0.0, DstIP: 92.204.128.155, SrcPort: 62416, DstPort: 4433, Protocol: tcp, IngressInterface: inside, EgressInterface: outside, IngressZone: inside, EgressZone: outside, IngressVRF: Global, EgressVRF: Global, ACPolicy: Pol_Access_Control, AccessControlRuleName: Allow_URLs, Prefilter Policy: Default Prefilter Policy_1, Client: SSL client, ApplicationProtocol: HTTPS, ConnectionDuration: 98, InitiatorPackets: 14, ResponderPackets: 16, InitiatorBytes: 1802, ResponderBytes: 6133, NAPPolicy: Balanced Security and Connectivity, SSLPolicy: None, SSLFlowStatus: Success, SSLCipherSuite: Unknown, SSLCertificate: d08c844c3b312370e87118a194ae6f8d52d3626b, SSLVersion: Unknown, SSLServerCertStatus: Valid, SSLActualAction: Do Not Decrypt, SSLExpectedAction: Do Not Decrypt, SSLServerName: ns1002325.ip-92-204-128.us, URLCategory: Infrastructure and Content Delivery Networks, URLReputation: Neutral, URL: https://ns1002325.ip-92-204-128.us","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-02-01T19:42:13.419+0530","log.logger":"syslog","log.origin":{"file.name":"syslog/input.go","file.line":285},"message":"can't parse event as syslog rfc3164","service.name":"filebeat","message":"Jan 24 2024 11:22:41 : %FTD-6-430003: EventPriority: Low, DeviceUUID: 70278058-1b77-11ec-a090-feeeb3702f49, InstanceID: 4, FirstPacketSecond: 2024-01-24T11:21:03Z, ConnectionID: 40428, AccessControlRuleAction: Allow, SrcIP: 0.0.0.0, DstIP: 92.204.128.155, SrcPort: 62416, DstPort: 4433, Protocol: tcp, IngressInterface: inside, EgressInterface: outside, IngressZone: inside, EgressZone: outside, IngressVRF: Global, EgressVRF: Global, ACPolicy: Pol_Access_Control, AccessControlRuleName: Allow_URLs, Prefilter Policy: Default Prefilter Policy_1, Client: SSL client, ApplicationProtocol: HTTPS, ConnectionDuration: 98, InitiatorPackets: 14, ResponderPackets: 16, InitiatorBytes: 1802, ResponderBytes: 6133, NAPPolicy: Balanced Security and Connectivity, SSLPolicy: None, SSLFlowStatus: Success, SSLCipherSuite: Unknown, SSLCertificate: d08c844c3b312370e87118a194ae6f8d52d3626b, SSLVersion: Unknown, SSLServerCertStatus: Valid, SSLActualAction: Do Not Decrypt, SSLExpectedAction: Do Not Decrypt, SSLServerName: ns1002325.ip-92-204-128.us, URLCategory: Infrastructure and Content Delivery Networks, URLReputation: Neutral, URL: https://ns1002325.ip-92-204-128.us","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-02-01T19:42:14.421+0530","log.logger":"syslog","log.origin":{"file.name":"syslog/input.go","file.line":285},"message":"can't parse event as syslog rfc3164","service.name":"filebeat","message":"Jan 24 2024 11:22:41 : %FTD-6-430003: EventPriority: Low, DeviceUUID: 70278058-1b77-11ec-a090-feeeb3702f49, InstanceID: 4, FirstPacketSecond: 2024-01-24T11:21:03Z, ConnectionID: 40428, AccessControlRuleAction: Allow, SrcIP: 0.0.0.0, DstIP: 92.204.128.155, SrcPort: 62416, DstPort: 4433, Protocol: tcp, IngressInterface: inside, EgressInterface: outside, IngressZone: inside, EgressZone: outside, IngressVRF: Global, EgressVRF: Global, ACPolicy: Pol_Access_Control, AccessControlRuleName: Allow_URLs, Prefilter Policy: Default Prefilter Policy_1, Client: SSL client, ApplicationProtocol: HTTPS, ConnectionDuration: 98, InitiatorPackets: 14, ResponderPackets: 16, InitiatorBytes: 1802, ResponderBytes: 6133, NAPPolicy: Balanced Security and Connectivity, SSLPolicy: None, SSLFlowStatus: Success, SSLCipherSuite: Unknown, SSLCertificate: d08c844c3b312370e87118a194ae6f8d52d3626b, SSLVersion: Unknown, SSLServerCertStatus: Valid, SSLActualAction: Do Not Decrypt, SSLExpectedAction: Do Not Decrypt, SSLServerName: ns1002325.ip-92-204-128.us, URLCategory: Infrastructure and Content Delivery Networks, URLReputation: Neutral, URL: https://ns1002325.ip-92-204-128.us","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-02-01T19:42:15.424+0530","log.logger":"syslog","log.origin":{"file.name":"syslog/input.go","file.line":285},"message":"can't parse event as syslog rfc3164","service.name":"filebeat","message":"Jan 24 2024 11:22:41 : %FTD-6-430003: EventPriority: Low, DeviceUUID: 70278058-1b77-11ec-a090-feeeb3702f49, InstanceID: 4, FirstPacketSecond: 2024-01-24T11:21:03Z, ConnectionID: 40428, AccessControlRuleAction: Allow, SrcIP: 0.0.0.0, DstIP: 92.204.128.155, SrcPort: 62416, DstPort: 4433, Protocol: tcp, IngressInterface: inside, EgressInterface: outside, IngressZone: inside, EgressZone: outside, IngressVRF: Global, EgressVRF: Global, ACPolicy: Pol_Access_Control, AccessControlRuleName: Allow_URLs, Prefilter Policy: Default Prefilter Policy_1, Client: SSL client, ApplicationProtocol: HTTPS, ConnectionDuration: 98, InitiatorPackets: 14, ResponderPackets: 16, InitiatorBytes: 1802, ResponderBytes: 6133, NAPPolicy: Balanced Security and Connectivity, SSLPolicy: None, SSLFlowStatus: Success, SSLCipherSuite: Unknown, SSLCertificate: d08c844c3b312370e87118a194ae6f8d52d3626b, SSLVersion: Unknown, SSLServerCertStatus: Valid, SSLActualAction: Do Not Decrypt, SSLExpectedAction: Do Not Decrypt, SSLServerName: ns1002325.ip-92-204-128.us, URLCategory: Infrastructure and Content Delivery Networks, URLReputation: Neutral, URL: https://ns1002325.ip-92-204-128.us","ecs.version":"1.6.0"}
I performed this test since I saw this issue in the production environment.
Looking for a solution.
Thanks,
Amol