[Filebeat][Checkpoint module] data stream timestamp field [@timestamp] is missing

Elastic Stack Beats
1

Hi,

I'm trying to ingest CheckPoint native Syslog exports of security gateway (firewall) logs. My understanding is that integration was previously via CEF, which did not pass through sufficient detail, but that the native syslog format was merged here: Checkpoint Syslog Filebeat module by P1llus · Pull Request #17682 · elastic/beats · GitHub

We had the following problem with CheckPoint R81 and continue to experience the same problem with the latest generally recommended version R81.10. We have configured the CheckPoint log exporter via SmartConsole, as follows:
image

Format is set as standard 'Syslog' format, which should include all the additional CheckPoint fields:
image

The problem we experiencing is that nothing is actually ingested, we receive the following error:

image
image1150×529 113 KB

The input pipeline was automatically configured when we added the Check Point module to an Elastic Agent via Fleet. This input pipeline appears to refer to fields which Check Point don't appear to generate:

image
image532×714 33.7 KB

CheckPoint documentation for the description of fields in Check Point Logs does not include '@timestamp' or 'timestamp':
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk144192

We are running Elastic Stack:

  • Version: 8.3.2
  • Operating System: Debian 11

Regards
David Herselman

2

Hi @bbs2web - welcome to the Discuss community and thanks for raising this issue. We have yet to formally test our integration with R81.10 but certainly have plans to do so.

Could you share the raw events with us, so we can determine the format of the timestamp which Log Exporter is generating (please ensure any sensitive information within the event is removed). @taylor-swanson could you please take a look once we have a sample?

3

Hi Jamie,

I have a packet capture, if that helps but the CheckPoint documentation 'Description of Fields in Check Point Logs' appears to break things down in the 'Security Logs' section, particularly the 'Security Gateway - Firewall Fields' section of the documentation.

Is there a way I can upload a pcap without this become available to anyone and everyone in the forums?

Regards
David Herselman

4

Apologies, re-read your message. I exported one of the log files to CSV (more like semi-colon separated) and then scrubbed a bunch of logs. I don't appear to see a way of attaching a file so can only paste a couple here now:

num;date;time;orig;type;action;alert;i/f_name;i/f_dir;product;LogId;ContextNum;origin_id;ContentVersion;hll_key;SequenceNum;log_sys_message;ProductFamily;inzone;outzone;service_id;src;dst;proto;xlatesrc;xlatedst;NAT_rulenum;NAT_addtnl_rulenum;needs_browse_time;security_inzone;security_outzone;protocol;sig_id;user;src_user_name;src_machine_name;src_user_dn;snid;dst_user_name;dst_machine_name;dst_user_dn;UP_match_table_match_id;UP_match_table_layer_uuid;UP_match_table_layer_name;UP_match_table_rule_uid;UP_match_table_rule_name;tls_server_host_name;sni;certificate_validity;dst_dynobj_name;dst_domain_name;dst_uo_name;dst_uo_icon;dst_object_type;NAT_rule_uid;NAT_addtnl_rule_uid;context_num;service;s_port;xlatedport;xlatesport;duration;last_hit_time;update_count;creation_time;connection_count;aggregated_log_count;web_client_type;web_server_type;UP_match_2_app_table_match_id;UP_match_2_app_table_app_id;UP_primary_app_table_primary_app;UP_app_table_id;UP_app_table_name;UP_app_table_app_desc;UP_app_table_category;UP_app_table_matched_category;UP_app_table_properties;UP_app_table_risk;UP_app_table_sig_id;resource_table_resource;resource_table_method;referrer;user_agent;Log delay;src_dynobj_name;src_domain_name;src_uo_name;src_uo_icon;src_object_type;UP_action_table_action;rule_guid;hit;policy;first_hit_time;log_id;start_time;segment_time;elapsed;packets;bytes;client_inbound_packets;client_outbound_packets;server_inbound_packets;server_outbound_packets;client_inbound_bytes;client_outbound_bytes;server_inbound_bytes;server_outbound_bytes;client_inbound_interface;client_outbound_interface;server_inbound_interface;server_outbound_interface;Unauthorized_SNI;browse_time;UP_urlf_table_id;UP_urlf_table_name;UP_urlf_table_app_desc;UP_urlf_table_category;UP_urlf_table_matched_category;UP_urlf_table_properties;UP_urlf_table_risk;UP_urlf_table_sig_id;connection_luuid;status;short_desc;long_desc;scan_hosts_hour;scan_hosts_day;scan_hosts_week;unique_detected_hour;unique_detected_day;unique_detected_week;scan_mail;url_count;rule;rule_uid;rule_name;sub_policy_name;sub_policy_uid;email_control;email_session_id;information;email_id;from;to;email_recipients_num;reason;TCP packet out of state;tcp_flags;client_type_os;action_reason;ICMP;ICMP Type;ICMP Code;message_info;attack;Attack Info;Protection Name;Protection ID;Severity;Confidence Level;Industry Reference;Performance Impact;Protection Type;Description URL;packet_capture_unique_id;packet_capture_time;packet_capture_name;SmartDefense profile;policy_time;session_id;Source_OS;dst_country;malware_rule_id;malware_rule_name;resource;reject_id_kid;ser_agent_kid;server_kid;TP_match_table_layer_uuid;TP_match_table_layer_name;TP_match_table_malware_rule_id;TP_match_table_malware_rule_name;TP_match_table_SmartDefense profile;contract_name;db_ver;subs_exp;description;Update Status;subscription_stat;subscription_stat_desc;next_update_desc;client_name;client_version;client_build;domain_name;host_type;os_name;os_version;os_edition;os_service_pack;os_build;os_bits;browser;endpoint_ip;device_identification;latitude;longitude;MACSourceAddress;auth_status;identity_src;src_user_group;src_machine_group;auth_method;identity_type;Authentication trial;roles;version;comment;update_service;Suppressed logs;sent_bytes;received_bytes;certificate_resource;certificate_validation;failure_impact;termination_reason;fw_message;proxy_src_ip;http_location;content_type;content_disposition;requested_with;via;http_server;content_length;method;http_status;authorization;http_host;protocol_name;protection_id;Streaming Engine;rpc_prog;srckeyid;dstkeyid;encryption failure:;peer gateway;scheme:;methods:;reject_category;fw_subproduct;vpn_feature_name;dynamic object;change type;modify type;ip ranges;IKE:;CookieI;CookieR;msgid;IKE notification:;Certificate DN:;IKE IDs:;partner;community;system_application;cp_component_name;cp_component_version;package_action;operation_results;cvpn_category;event_type;auth_method2;auth_method3;login_option;failed_login_factor;failed_login_factor_num;user_dn;fingerprint;certificate_serial_number;certificate_issuer;user_group;hardware_model;session_timeout;login_timestamp;host_ip;office_mode_ip;tunnel_protocol;license;Suppressed_Logs;More;session_uid;mac_address;Hostname;auth_encryption_methods;message;vpn_user;old IP;old port;new IP;new port;DCE-RPC Interface UUID;note;connection_uid;blade_name;control_log_type;file_name;sys_message:;c_bytes;Last Rematch Time
0;17Jul2022;0:00:47;100.127.202.23;control; ;;daemon;inbound;VPN-1 & FireWall-1;-1;-1;CN=Company_Log,O=Company_Server_1.company.com.gbu7jf;5;18446744073709551615;1;Log file has been switched to: 2022-07-17_000000.log;Network;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
1;16Jul2022;23:59:59;192.0.2.90;connection;accept;;eth0.11;inbound;VPN-1 & FireWall-1;0;1;CN=gw1,O=Company_Server_1.company.com.gbu7jf;5;16666636505222387524;11;;Network;Internal;External;https;192.168.17.10;192.0.65.26;tcp;192.0.2.231;;54;0;1;InternalZone;;HTTPS;4;Joe Doe (joed)(+);Joe Doe (joed)(+);lt-joed@ad.company.com;CN=Joe Doe,OU=Users,OU=Company,DC=ad,DC=company,DC=com(+);;;server18@ad.company.com;;48,16777269;eb785a20-4294-48fe-9226-165660243e7f,3984cc0a-aa85-49aa-8b71-ac8736a162fb;Network,Application;e37a79cb-a37e-4de9-8a92-88bee71637d5,78e32861-3088-4576-8f45-e6a486277b29;9.40_._._Allow identified users,9.45_._._Cleanup rule;fluffy.company.com;fluffy.company.com;Trusted;;;Timbuktu;@app/cp_geo_ml;;849c7739-2249-4b75-b86a-7a44c1a3c2c0;;1;443;49240;;37732;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
2;16Jul2022;23:59:59;192.0.2.90;account;accept;;eth0.10;outbound;VPN-1 & FireWall-1;6;1;CN=gw1,O=Company_Server_1.company.com.gbu7jf;5;6056588561559531162;8;;Network;;;https_10051;192.168.20.14;192.0.65.26;tcp;192.0.2.231;;54;0;1;InternalZone;;HTTPS;4;;;laptop-scoutm@ad.company.com;;;;server18@ad.company.com;;40,16777239;eb785a20-4294-48fe-9226-165660243e7f,3984cc0a-aa85-49aa-8b71-ac8736a162fb;Network,Application;7de163cb-dc42-4fb8-9559-9c4988c53d18,5cf233d5-57f3-4745-ae4c-61e1af2bbd4b;9.32_._._Do not require authentication,9.15_._._Unauthenticated - Fluffy;192.0.65.26;;Untrusted;;;Timbuktu;@app/cp_geo_ml;;849c7739-2249-4b75-b86a-7a44c1a3c2c0;;1;10051;33722;;37733;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;16Jul2022 23:59:59;16Jul2022 23:59:59;0:00:00;30;6129;12;18;9;24;2590;3539;3539;2590;eth0.11;;;eth0.10;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
3;16Jul2022;23:59:59;192.0.2.90;connection;accept;;eth0.11;inbound;VPN-1 & FireWall-1;0;1;CN=gw1,O=Company_Server_1.company.com.gbu7jf;5;3250452641575952679;13;;Network;Internal;External;http;192.168.1.17;104.18.32.68;tcp;192.0.2.231;;54;0;;InternalZone;;HTTP;0;;;mixmaster@ad.company.com;;;;;;40,16777245;eb785a20-4294-48fe-9226-165660243e7f,3984cc0a-aa85-49aa-8b71-ac8736a162fb;Network,Application;7de163cb-dc42-4fb8-9559-9c4988c53d18,7446889b-e12d-4c9b-b839-70bb9c8d488f;9.32_._._Do not require authentication,9.21_._._Allow - Unauthenticated;;;;;;;;;849c7739-2249-4b75-b86a-7a44c1a3c2c0;;1;80;36497;;27561;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
4;16Jul2022;23:59:59;192.0.2.90;connection;accept;;eth0.11;outbound;VPN-1 & FireWall-1;4;2;CN=gw1,O=Company_Server_1.company.com.gbu7jf;5;7361627382187237226;14;;Network;Internal;External;http;192.168.1.17;104.18.32.68;tcp;192.0.2.231;;54;0;;InternalZone;;HTTP;0;;;mixmaster@ad.company.com;;;;;;40,16777245;eb785a20-4294-48fe-9226-165660243e7f,3984cc0a-aa85-49aa-8b71-ac8736a162fb;Network,Application;7de163cb-dc42-4fb8-9559-9c4988c53d18,7446889b-e12d-4c9b-b839-70bb9c8d488f;9.32_._._Do not require authentication,9.21_._._Allow - Unauthenticated;;;;;;;;;849c7739-2249-4b75-b86a-7a44c1a3c2c0;;2;80;36497;;27561;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
5;16Jul2022;23:59:59;192.0.2.90;session;accept;;eth0.11;inbound;Application Control;352;-1;CN=gw1,O=Company_Server_1.company.com.gbu7jf;5;3250452641575952679;2;;Network;Internal;External;http;192.168.1.17;104.18.32.68;tcp;;;;;;;;HTTP;0;;;mixmaster@ad.company.com;;;;;;40,16777245;eb785a20-4294-48fe-9226-165660243e7f,3984cc0a-aa85-49aa-8b71-ac8736a162fb;Network,Application;7de163cb-dc42-4fb8-9559-9c4988c53d18,7446889b-e12d-4c9b-b839-70bb9c8d488f;9.32_._._Do not require authentication,9.21_._._Allow - Unauthenticated;;;;;;;;;;;;80;;;;3:00:00;17Jul2022  1:13:08;4;16Jul2022 23:59:59;2;3;Other: Microsoft-CryptoAPI/10.0;;16777245;60529026;60529026;60529026;Windows 10 Update;Windows 10 OS network traffic, it happen usually by update or upgrade.;Network Protocols;Network Protocols;Encrypts communications, Low Risk, Network Protocols;2;60529026:2;http://ocsp.sectigo.com/MFIwUDBOMEwwSjAJBgUrDgMCGgUABBRDC9IOTxN6GmyRjyTl2n4yTUczyAQUjYxexFStiuF36Zv5mwXhuAGNYeECEQDwhJzwIxXNs9fdoihwqMCx;GET;;;;;;;;;;;;;;;;;;14;1664;6;8;4;13;524;1140;1140;564;;;;;;;;;;;;;;;;;;;;;;;;;;1;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
6;16Jul2022;23:59:59;192.0.2.90;session;accept;;eth0.11;outbound;Application Control;352;-1;CN=gw1,O=Company_Server_1.company.com.gbu7jf;5;7361627382187237226;3;;Network;;;http;192.168.1.17;104.18.32.68;tcp;;;;;;;;HTTP;0;;;mixmaster@ad.company.com;;;;;;40,16777245;eb785a20-4294-48fe-9226-165660243e7f,3984cc0a-aa85-49aa-8b71-ac8736a162fb;Network,Application;7de163cb-dc42-4fb8-9559-9c4988c53d18,7446889b-e12d-4c9b-b839-70bb9c8d488f;9.32_._._Do not require authentication,9.21_._._Allow - Unauthenticated;;;;;;;;;;;;80;;;;3:00:00;17Jul2022  0:01:04;3;16Jul2022 23:59:59;1;2;Other: Microsoft-CryptoAPI/10.0;;16777245;10075086;10075086;10075086;OCSP;The Online Certificate Status Protocol (OCSP) is an Internet protocol used for obtaining the revocation status of an X.509 digital certificate. It was created as an alternative to certificate revocation lists.;Network Protocols;Network Protocols;Encrypts communications, Very Low Risk, Network Protocols;1;10075086:35;http://ocsp.sectigo.com/MFIwUDBOMEwwSjAJBgUrDgMCGgUABBRDC9IOTxN6GmyRjyTl2n4yTUczyAQUjYxexFStiuF36Zv5mwXhuAGNYeECEQDwhJzwIxXNs9fdoihwqMCx;GET;;Microsoft-CryptoAPI/10.0;;;;;;;;;;;;;;;;14;1661;6;8;4;12;524;1137;1137;524;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
7;16Jul2022;23:59:59;192.0.2.90;connection;accept;;eth0.10;outbound;VPN-1 & FireWall-1;0;1;CN=gw1,O=Company_Server_1.company.com.gbu7jf;5;6353774620623721536;17;;Network;;;SSH_2200;192.168.20.11;192.0.1.39;tcp;192.0.2.231;;54;0;;InternalZone;;SSH2;1;;;core1@ad.company.com;;;;;;40,16777245;eb785a20-4294-48fe-9226-165660243e7f,3984cc0a-aa85-49aa-8b71-ac8736a162fb;Network,Application;7de163cb-dc42-4fb8-9559-9c4988c53d18,7446889b-e12d-4c9b-b839-70bb9c8d488f;9.32_._._Do not require authentication,9.21_._._Allow - Unauthenticated;;;;;;Timbuktu;@app/cp_geo_ml;;849c7739-2249-4b75-b86a-7a44c1a3c2c0;;1;2200;42222;;37734;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
8;16Jul2022;23:59:59;192.0.2.90;account;accept;;eth0.10;outbound;VPN-1 & FireWall-1;6;1;CN=gw1,O=Company_Server_1.company.com.gbu7jf;5;3920171494212229393;5;;Network;;;https_10051;192.168.1.20;192.0.65.26;tcp;192.0.2.231;;54;0;1;InternalZone;;HTTPS;4;;;connect@ad.company.com;;;;server18@ad.company.com;;40,16777239;eb785a20-4294-48fe-9226-165660243e7f,3984cc0a-aa85-49aa-8b71-ac8736a162fb;Network,Application;7de163cb-dc42-4fb8-9559-9c4988c53d18,5cf233d5-57f3-4745-ae4c-61e1af2bbd4b;9.32_._._Do not require authentication,9.15_._._Unauthenticated - Fluffy;192.0.65.26;;Untrusted;;;Timbuktu;@app/cp_geo_ml;;849c7739-2249-4b75-b86a-7a44c1a3c2c0;;1;10051;59649;;37735;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;16Jul2022 23:59:59;16Jul2022 23:59:59;0:00:00;37;7572;13;24;12;26;3993;3579;3579;3993;eth0.11;;;eth0.10;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

512 lines of logs available in the following archive:
https://file.io/4QcDv7XO36SN

5

Hi @bbs2web, apologies for getting back a little late, I didn't see the replies on this thread.

Unfortunately, I don't think CSV will help us here. We'll need the syslog-formatted log messages, since that is what the pipeline will be parsing (and the timestamp format also likely changes between export formats). I'm not familiar with Check Point's Log Exporter, so I don't know if there's a way to specify the syslog format and export to log at the same time. Otherwise, if you still have that pcap, you should be able to open it with something like Wireshark and copy and scrub the logs(s) from there.

6

Hi,

Many thanks for the guidance, I hope the 'message' from the failed syslog ingestion logs and pcap exports are in a better format now. I supplied them in an associated GitHub issue I'd opened here:

Please let me know if there is anything else I could assist with.

7

That looks great, thanks! The github issue will be a better place to discuss/track progress as well.

8

Sorry @bbs2web - I totally missed your reply last week. Thanks for providing those samples - they'll be a big help. If you want to share any additional samples, just let me know and I can provide you with a secure upload link, to avoid posting samples here.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.