Filebeat Cisco module can't parse 305011, 302015, 302013, or 722015 message types

Elastic Stack Beats
1

Version 7.9.2
Debian 10
Problematic file: /usr/share/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml

305011 parse pattern doesn't account for presence of source.user.name
pattern: "Built %{} %{network.transport} translation from %{temp.cisco.source_interface}:%{source.address}/%{source.port} to %{temp.cisco.destination_interface}:%{destination.address}/%{destination.port}"

Logstash does this correctly:
CISCOFW305011 %{CISCO_ACTION:action} %{CISCO_XLATE_TYPE:xlate_type} %{WORD:protocol} translation from %{DATA:src_interface}:%{IP:src_ip}(/%{INT:src_port})?((%{DATA:src_fwuser}))? to %{DATA:src_xlated_interface}:%{IP:src_xlated_ip}/%{DATA:src_xlated_port}

same issue for 302015, 302013
filebeat:
pattern: "Built %{network.direction} %{network.transport} connection %{temp.cisco.connection_id} for %{temp.cisco.source_interface}:%{source.address}/%{source.port} (%{temp.natsrcip}/%{temp.cisco.mapped_source_port}) to %{temp.cisco.destination_interface}:%{destination.address}/%{destination.port} (%{temp.natdstip}/%{temp.cisco.mapped_destination_port})"

Logstash again does it correctly:
CISCOFW302013_302014_302015_302016 %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port}( (%{IP:src_mapped_ip}/%{INT:src_mapped_port}))?((%{DATA:src_fwuser}))? to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}( (%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}))?((%{DATA:dst_fwuser}))?( duration %{TIME:duration} bytes %{INT:bytes})?(?: %{CISCO_REASON:reason})?( (%{DATA:user}))?

Also 722051 doesn't account for angle brackets around the IP addresses. Logstash doesn't parse 722051 messages at all.

Resulting anonymized error messages:
Oct 03 13:53:55 xxxx filebeat[10145]: 2020-10-03T13:53:55.643-0500 WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfd65010cf489145, ext:3086635459, loc:(*time.Location)(0x607ff60)}, Meta:{"pipeline":"filebeat-7.9.2-cisco-asa-asa-ftd-pipeline"}, Fields:{"agent":{"ephemeral_id":"0eb2fd0e-c07e-430c-ab52-af187d1350dd","hostname":"xxxx","id":"fa6227e2-086e-4131-8b98-74cd576212aa","name":"xxxx","type":"filebeat","version":"7.9.2"},"ecs":{"version":"1.5.0"},"event":{"dataset":"cisco.asa","module":"cisco","timezone":"-05:00"},"fileset":{"name":"asa"},"input":{"type":"log"},"log":{"file":{"path":"/path/file.log"},"offset":1457},"message":"Oct 3 04:00:14 xxxx %ASA-6-305011: Built dynamic UDP translation from any:10.190.132.108/35854(LOCAL\username) to VLANyyyy:aaa.bbb.ccc.ddd/35854","service":{"type":"cisco"},"tags":["cisco-asa","forwarded"]}, Private:file.State{Id:"native::22-65026", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000c3dc70), Source:"/path/file.log", Offset:1628, Timestamp:time.Time{wall:0xbfd650103ab4b607, ext:815143094, loc:(*time.Location)(0x607ff60)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x16, Device:0xfe02}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [source.port] of type [long] in document with id 'Gn3Q73QBS_mRngGNO7nm'. Preview of field's value: '35854(LOCAL\username)'","caused_by":{"type":"illegal_argument_exception","reason":"For input string: "35854(LOCAL\username)""}}

Oct 03 13:53:55 xxxx filebeat[10145]: 2020-10-03T13:53:55.643-0500 WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfd65010cf48eb55, ext:3086658514, loc:(*time.Location)(0x607ff60)}, Meta:{"pipeline":"filebeat-7.9.2-cisco-asa-asa-ftd-pipeline"}, Fields:{"agent":{"ephemeral_id":"0eb2fd0e-c07e-430c-ab52-af187d1350dd","hostname":"xxxx","id":"fa6227e2-086e-4131-8b98-74cd576212aa","name":"xxxx","type":"filebeat","version":"7.9.2"},"ecs":{"version":"1.5.0"},"event":{"dataset":"cisco.asa","module":"cisco","timezone":"-05:00"},"fileset":{"name":"asa"},"input":{"type":"log"},"log":{"file":{"path":"/path/file.log"},"offset":1628},"message":"Oct 3 04:00:14 yyyy %ASA-6-302015: Built inbound UDP connection 2571109148 for VLANzzz:aaa.bbb.ccc.ddd/35854 (eee.fff.ggg.hhh/35854)(LOCAL\username) to VLANyyy:ppp.ppp.ppp.ppp:53 (ppp.ppp.ppp.ppp/53) (username)","service":{"type":"cisco"},"tags":["cisco-asa","forwarded"]}, Private:file.State{Id:"native::22-65026", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000c3dc70), Source:"/path/file.log", Offset:1871, Timestamp:time.Time{wall:0xbfd650103ab4b607, ext:815143094, loc:(*time.Location)(0x607ff60)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x16, Device:0xfe02}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [source.nat.port] of type [long] in document with id 'HX3Q73QBS_mRngGNO7nm'. Preview of field's value: '35854)(LOCAL\\username'","caused_by":{"type":"illegal_argument_exception","reason":"For input string: "35854)(LOCAL\\username""}}

Oct 03 13:53:55 xxxx filebeat[10145]: 2020-10-03T13:53:55.729-0500 WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfd65010cf4fa6a2, ext:3087099667, loc:(*time.Location)(0x607ff60)}, Meta:{"pipeline":"filebeat-7.9.2-cisco-asa-asa-ftd-pipeline"}, Fields:{"agent":{"ephemeral_id":"0eb2fd0e-c07e-430c-ab52-af187d1350dd","hostname":"ixxxx","id":"fa6227e2-086e-4131-8b98-74cd576212aa","name":"xxxx","type":"filebeat","version":"7.9.2"},"ecs":{"version":"1.5.0"},"event":{"dataset":"cisco.asa","module":"cisco","timezone":"-05:00"},"fileset":{"name":"asa"},"input":{"type":"log"},"log":{"file":{"path":"/path/file.log"},"offset":206995887},"message":"Oct 3 00:12:59 yyyy%ASA-6-302013: Built inbound TCP connection 2568709940 for VLANyyy:aaa.bbb.ccc.ddd/57999 (hhh.iii.jjj.kkk/57999)(LOCAL\username) to VLANzzz:mmm.nnn.ooo.ppp/443 (/443) (username)","service":{"type":"cisco"},"tags":["cisco-asa","forwarded"]}, Private:file.State{Id:"native::15-65026", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000c3d5f0), Source:"/path/file.log", Offset:206996125, Timestamp:time.Time{wall:0xbfd650103acaa335, ext:816580032, loc:(*time.Location)(0x607ff60)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xf, Device:0xfe02}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [source.nat.port] of type [long] in document with id 'uH3Q73QBS_mRngGNPLlz'. Preview of field's value: '57999)(LOCAL\\username'","caused_by":{"type":"illegal_argument_exception","reason":"For input string: "57999)(LOCAL\\username""}}

Oct 03 13:54:16 xxxxfilebeat[10145]: 2020-10-03T13:54:16.769-0500 WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfd6501526b0742b, ext:20479319711, loc:(time.Location)(0x607ff60)}, Meta:{"pipeline":"filebeat-7.9.2-cisco-asa-asa-ftd-pipeline"}, Fields:{"agent":{"ephemeral_id":"0eb2fd0e-c07e-430c-ab52-af187d1350dd","hostname":"xxxx","id":"fa6227e2-086e-4131-8b98-74cd576212aa","name":"xxxx","type":"filebeat","version":"7.9.2"},"ecs":{"version":"1.5.0"},"event":{"dataset":"cisco.asa","module":"cisco","timezone":"-05:00"},"fileset":{"name":"asa"},"input":{"type":"log"},"log":{"file":{"path":"/path/file.log"},"offset":122047},"message":"Oct 3 06:00:14 yyy %ASA-4-722051: Group \u003cvpn-group-policy\u003e User \u003cusername*\u003e IP \u003caaa.bbb.ccc.ddd\u003e IPv4 Address \u003cggg.hhh.iii.jjj\u003e IPv6 address \u003c::\u003e assigned to session","service":{"type":"cisco"},"tags":["cisco-asa","forwarded"]}, Private:file.State{Id:"native::24-65026", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000d64000), Source:"/path/file.log", Offset:122224, Timestamp:time.Time{wall:0xbfd65010cf569f4a, ext:3087556555, loc:(*time.Location)(0x607ff60)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x18, Device:0xfe02}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [cisco.asa.assigned_ip] of type [ip] in document with id 'PwfQ73QB0kNeqwYQjoGm'. Preview of field's value: '<1aaa.bbb.ccc.ddd>'","caused_by":{"type":"illegal_argument_exception","reason":"'<aaa.bbb.ccc.ddd>' is not an IP string literal."}}

4

I crafted a grok pattern based on the Logstash pattern for 305011. Tested it in the Grok Debugger. It was fine. I put it into filebeat as the grok (not dissect) pattern for 305011 and get the exact same error. I know the changes are being recognized, because if I intentionally make a typo filebeat throws an error on the pipeline.

For reference, the last iteration of the pattern I tried was:

Built %{DATA} %{WORD:network.transport} translation from %{DATA:_temp_.cisco.source_interface}:%{IP:source.address}/%{POSINT:source.port}\(%{NOTSPACE:source.user.name}\) to %{DATA:_temp_.cisco.destination_interface}:%{IP:destination.address}/%{GREEDYDATA:destination.port}

5

Hi @rthielen, thanks for posting your solution here. Do you mind sharing some log lines as examples for us to add into the test folders in Filebeat for local testing? Thank you!

7

Well the above is not a solution. As I said, it worked in the Grok debugger, but it did not work in the asa-ftd-pipeline.yml pipeline configuration.

Here are sample log lines for each of the four messages with the IP addresses and usernames altered.

%ASA-6-305011 sample
Built dynamic UDP translation from any:111.111.132.108/35854(LOCAL\\username) to CVPN-USER-VLAN9999:000.000.121.232/35854

%ASA-6-302015 sample
Built inbound UDP connection 2571109148 for VLAN9999:111.111.132.108/35854 (000.000.121.232/35854)(LOCAL\\username) to VLAN9999:8.8.8.8/53 (8.8.8.8/53) (username)

%ASA-6-302013 sample
Built inbound TCP connection 2568709940 for VLAN9999:111.111.137.53/57999 (000.000.121.113/57999)(LOCAL\\username) to VLAN9999:111.111.7.60/443 (111.111.7.60/443) (username)

%ASA-4-722051 sample
Group <group-policy> User <username> IP <999.999.13.141> IPv4 Address <111.111.128.175> IPv6 address <::> assigned to session
8

If you look at the 32013 and 32015 messages and the errors in the first post, you will see that the dissect parser didn't stop parsing the source.nat.port when it encountered the closing parentheses. It continued to the next space.

9

This problem seems very similar to https://github.com/elastic/beats/pull/17964

10

Thank you!! @Mario_Castro Should we create a github issue for this?

12

Yeah, I think the best will be that @rthielen to create a new issue with some example log lines, linked to this post and wait for someone to have some time to improve the dissect parser or change that error code from dissect to Grok.

13

I have done so. https://github.com/elastic/beats/issues/21658

Note that I typo'd the title of this thread. I mentioned message 722015, when the text makes it clear I meant 722051.

14

I believe that I have a suggested fix for the issue. When I tried to fix the patterns before, I didn't realize that I needed to specify
-E filebeat.overwrite_pipelines=true
on the "filebeat setup --pipelines" command. The following diff output shows the patterns that worked for me.

diff /usr/share/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml.orig 
/usr/share/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml
297c297
<   - dissect:
---
>   - grok:
300c300,303
<       pattern: "Built %{network.direction} %{network.transport} connection %{_temp_.cisco.connection_id} for %{_temp_.cisco.source_interface}:%{source.address}/%{source.port} (%{_temp_.natsrcip}/%{_temp_.cisco.mapped_source_port}) to %{_temp_.cisco.destination_interface}:%{destination.address}/%{destination.port} (%{_temp_.natdstip}/%{_temp_.cisco.mapped_destination_port})"
---
>       patterns:
>       - Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:_temp_.cisco.connection_id} for %{DATA:_temp_.cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{IPORHOST:_temp_.natsrcip}/%{INT:_temp_.cisco.mapped_source_port}\)\(%{NOTSPACE:source.user.name}\) to %{DATA:_temp_.cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{IPORHOST:_temp_.natdstip}/%{INT:_temp_.cisco.mapped_destination_port}\) \(%{DATA:_temp_.cisco.source_username}\)
>       - Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:_temp_.cisco.connection_id} for %{DATA:_temp_.cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{IPORHOST:_temp_.natsrcip}/%{INT:_temp_.cisco.mapped_source_port}\) to %{DATA:_temp_.cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{IPORHOST:_temp_.natdstip}/%{INT:_temp_.cisco.mapped_destination_port}\) \(%{DATA:_temp_.cisco.source_username}\)
>       - Built %{WORD:network.direction} %{WORD:network.transport} connection %{INT:_temp_.cisco.connection_id} for %{DATA:_temp_.cisco.source_interface}:%{IPORHOST:source.address}/%{INT:source.port} \(%{IPORHOST:_temp_.natsrcip}/%{INT:_temp_.cisco.mapped_source_port}\) to %{DATA:_temp_.cisco.destination_interface}:%{IPORHOST:destination.address}/%{INT:destination.port} \(%{IPORHOST:_temp_.natdstip}/%{INT:_temp_.cisco.mapped_destination_port}\)
334c337
<       pattern: "Built %{} %{network.transport} translation from %{_temp_.cisco.source_interface}:%{source.address}/%{source.port} to %{_temp_.cisco.destination_interface}:%{destination.address}/%{destination.port}"
---
>       pattern: "Built %{} %{network.transport} translation from %{_temp_.cisco.source_interface}:%{source.address}/%{source.port}(%{source.user.name}) to %{_temp_.cisco.destination_interface}:%{destination.address}/%{destination.port}"
550c553
<       pattern: "Group %{} User %{source.user.name} IP %{source.address} IPv4 Address % {_temp_.cisco.assigned_ip} %{}"
---
>       pattern: "Group <%{}> User <%{source.user.name}> IP <%{source.address}> IPv4 Address <%{_temp_.cisco.assigned_ip}> %{}"
15

@rthielen Thanks for sharing the updated file. I also facing same issue and opened in discussion almost a month ago.. when i am trying to execute --pipelines to elastic directly to update this configuration getting following error.

filebeat setup --pipelines --modules cisco -E filebeat.overwrite_pipelines=true
Exiting: Error getting pipeline for fileset cisco/asa: Error YAML decoding the pipeline file: ../shared/ingest/asa-ftd-pipeline.yml: yaml: line 325: found unknown escape character

could you help me on this

16

Even I tried downgrading filebeat to 7.9.2 with updated 3ef415d file, still i am seeing this parsing error

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.