Where is my data of fortinet?

Elastic Stack Elasticsearch
21

@morad_della3
Is there a filebeat log about listening on UDP port?
Since you do not show me the logs I can not help...

Did you check to see if you are getting the FW logs?

You should see a filebeat log line like

{"log.level":"info","@timestamp":"2022-07-31T11:11:13.340-0700","log.origin":{"file.name":"udp/input.go","file.line":98},"message":"Starting UDP input","service.name":"filebeat","ecs.version":"1.6.0"}```
22 
root@elkfiras:/home/elkfiras# filebeat -e
{"log.level":"info","@timestamp":"2022-07-31T18:04:35.833Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:35.833Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: 18f18c9b-6051-49ef-a4ce-e9cce3299b83","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-31T18:04:38.838Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.845Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.846Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1076},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"18f18c9b-6051-49ef-a4ce-e9cce3299b83"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.846Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1085},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"45f722f492dcf1d13698c6cf618b339b1d4907be","libbeat":"8.3.2","time":"2022-07-06T10:12:50.000Z","version":"8.3.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.846Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1088},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":2,"version":"go1.18.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.850Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-07-31T00:55:47Z","containerized":false,"name":"elkfiras","ip":["127.0.0.1/8","::1/128","192.168.37.2/24","fe80::20c:29ff:fe65:82ff/64"],"kernel_version":"5.15.0-41-generic","mac":["00:0c:29:65:82:ff"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04 (Jammy Jellyfish)","major":22,"minor":4,"patch":0,"codename":"jammy"},"timezone":"UTC","timezone_offset_sec":0,"id":"24ec3a89c2b5417b9c2a0e9755bb64bc"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.854Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1121},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/home/elkfiras","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":13076,"ppid":12876,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-07-31T18:04:35.400Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.855Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: filebeat; Version: 8.3.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-31T18:04:38.876Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.877Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://192.168.37.2:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.881Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: elkfiras","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.884Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.885Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":143},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.886Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.916Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=1476","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.917Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.917Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.918Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.921Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 11337388005444501392)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.921Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.921Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":111},"message":"Input filestream starting","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:38.922Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:41.841Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:48.926Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: fortinet (firewall)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:48.931Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://192.168.37.2:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:48.996Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":162},"message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:48.996Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":173},"message":"CA certificate matching 'ca_trusted_fingerprint' found, adding it to 'certificate_authorities'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:49.001Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.3.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:49.025Z","log.origin":{"file.name":"udp/input.go","file.line":99},"message":"Starting UDP input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:49.027Z","log.logger":"UDP","log.origin":{"file.name":"dgram/server.go","file.line":99},"message":"Started listening for UDP connection","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:51.685Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(https://192.168.37.2:9200))","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:51.734Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":162},"message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:51.736Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":173},"message":"CA certificate matching 'ca_trusted_fingerprint' found, adding it to 'certificate_authorities'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:51.742Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.3.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:51.747Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.3.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:51.747Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":231},"message":"Auto ILM enable success.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:51.752Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":128},"message":"ILM policy filebeat successfully created.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:51.752Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":366},"message":"Set settings.index.lifecycle.name in template to {filebeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:51.812Z","log.logger":"template","log.origin":{"file.name":"template/load.go","file.line":245},"message":"Existing template will be overwritten, as overwrite is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:54.015Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":159},"message":"Try loading template filebeat-8.3.2 to Elasticsearch","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:54.111Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":127},"message":"Template with name \"filebeat-8.3.2\" loaded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:54.114Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":143},"message":"Data stream with name \"filebeat-8.3.2\" already exists.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:54.114Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":267},"message":"Loaded index template.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:04:54.132Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(https://192.168.37.2:9200)) established","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:05:08.899Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"id":"session-33.scope"},"memory":{"id":"session-33.scope","mem":{"usage":{"bytes":169803776}}}},"cpu":{"system":{"ticks":640,"time":{"ms":640}},"total":{"ticks":3130,"time":{"ms":3130},"value":0},"user":{"ticks":2490,"time":{"ms":2490}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":38},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","name":"filebeat","uptime":{"ms":33208},"version":"8.3.2"},"memstats":{"gc_next":65224120,"memory_alloc":35246944,"memory_sys":113823768,"memory_total":324133896,"rss":160706560},"runtime":{"goroutines":109}},"filebeat":{"events":{"added":10,"done":10},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":2},"output":{"events":{"acked":10,"active":0,"batches":3,"total":10},"read":{"bytes":40198},"type":"elasticsearch","write":{"bytes":390017}},"pipeline":{"clients":26,"events":{"active":0,"published":10,"retry":7,"total":10},"queue":{"acked":10,"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0.15,"15":0.16,"5":0.15,"norm":{"1":0.075,"15":0.08,"5":0.075}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:05:38.898Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":225280}}}},"cpu":{"system":{"ticks":670,"time":{"ms":30}},"total":{"ticks":3200,"time":{"ms":70},"value":0},"user":{"ticks":2530,"time":{"ms":40}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":38},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":63214},"version":"8.3.2"},"memstats":{"gc_next":65224120,"memory_alloc":36253152,"memory_sys":262144,"memory_total":325140104,"rss":160706560},"runtime":{"goroutines":109}},"filebeat":{"events":{"added":4,"done":4},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":4,"active":0,"batches":3,"total":4},"read":{"bytes":1333},"write":{"bytes":5837}},"pipeline":{"clients":26,"events":{"active":0,"published":4,"total":4},"queue":{"acked":4}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.09,"15":0.15,"5":0.13,"norm":{"1":0.045,"15":0.075,"5":0.065}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:06:08.894Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":-77824}}}},"cpu":{"system":{"ticks":700,"time":{"ms":30}},"total":{"ticks":3260,"time":{"ms":60},"value":0},"user":{"ticks":2560,"time":{"ms":30}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":93205},"version":"8.3.2"},"memstats":{"gc_next":65224120,"memory_alloc":37150488,"memory_total":326037440,"rss":160960512},"runtime":{"goroutines":107}},"filebeat":{"events":{"added":6,"done":6},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":6,"active":0,"batches":5,"total":6},"read":{"bytes":2078},"write":{"bytes":8939}},"pipeline":{"clients":26,"events":{"active":0,"published":6,"total":6},"queue":{"acked":6}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.05,"15":0.15,"5":0.12,"norm":{"1":0.025,"15":0.075,"5":0.06}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:06:38.895Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":114688}}}},"cpu":{"system":{"ticks":750,"time":{"ms":50}},"total":{"ticks":3340,"time":{"ms":80},"value":0},"user":{"ticks":2590,"time":{"ms":30}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":123206},"version":"8.3.2"},"memstats":{"gc_next":65224120,"memory_alloc":38257656,"memory_total":327144608,"rss":160960512},"runtime":{"goroutines":107}},"filebeat":{"events":{"added":6,"done":6},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":6,"active":0,"batches":6,"total":6},"read":{"bytes":2236},"write":{"bytes":9318}},"pipeline":{"clients":26,"events":{"active":0,"published":6,"total":6},"queue":{"acked":6}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.03,"15":0.14,"5":0.11,"norm":{"1":0.015,"15":0.07,"5":0.055}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:07:08.895Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":-53768192}}}},"cpu":{"system":{"ticks":830,"time":{"ms":80}},"total":{"ticks":3490,"time":{"ms":150},"value":0},"user":{"ticks":2660,"time":{"ms":70}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":153206},"version":"8.3.2"},"memstats":{"gc_next":22086632,"memory_alloc":10684008,"memory_total":328294944,"rss":107094016},"runtime":{"goroutines":107}},"filebeat":{"events":{"active":1,"added":8,"done":7},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":7,"active":0,"batches":6,"total":7},"read":{"bytes":2448},"write":{"bytes":10989}},"pipeline":{"clients":26,"events":{"active":1,"published":8,"total":8},"queue":{"acked":7}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.09,"15":0.14,"5":0.11,"norm":{"1":0.045,"15":0.07,"5":0.055}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:07:38.898Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":12288}}}},"cpu":{"system":{"ticks":880,"time":{"ms":50}},"total":{"ticks":3560,"time":{"ms":70},"value":0},"user":{"ticks":2680,"time":{"ms":20}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":183214},"version":"8.3.2"},"memstats":{"gc_next":22086632,"memory_alloc":12053000,"memory_total":329663936,"rss":107094016},"runtime":{"goroutines":107}},"filebeat":{"events":{"active":-1,"added":8,"done":9},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":9,"active":0,"batches":7,"total":9},"read":{"bytes":3037},"write":{"bytes":13224}},"pipeline":{"clients":26,"events":{"active":0,"published":8,"total":8},"queue":{"acked":9}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.19,"15":0.15,"5":0.13,"norm":{"1":0.095,"15":0.075,"5":0.065}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:08:08.891Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":910,"time":{"ms":30}},"total":{"ticks":3620,"time":{"ms":60},"value":0},"user":{"ticks":2710,"time":{"ms":30}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":213204},"version":"8.3.2"},"memstats":{"gc_next":22086632,"memory_alloc":12780992,"memory_total":330391928,"rss":107094016},"runtime":{"goroutines":107}},"filebeat":{"events":{"added":5,"done":5},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":5,"active":0,"batches":3,"total":5},"read":{"bytes":1550},"write":{"bytes":7280}},"pipeline":{"clients":26,"events":{"active":0,"published":5,"total":5},"queue":{"acked":5}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.11,"15":0.14,"5":0.12,"norm":{"1":0.055,"15":0.07,"5":0.06}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:08:38.895Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":-45056}}}},"cpu":{"system":{"ticks":950,"time":{"ms":40}},"total":{"ticks":3700,"time":{"ms":80},"value":0},"user":{"ticks":2750,"time":{"ms":40}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":243212},"version":"8.3.2"},"memstats":{"gc_next":22086632,"memory_alloc":13930536,"memory_total":331541472,"rss":107094016},"runtime":{"goroutines":107}},"filebeat":{"events":{"added":5,"done":5},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":5,"active":0,"batches":5,"total":5},"read":{"bytes":1862},"write":{"bytes":7755}},"pipeline":{"clients":26,"events":{"active":0,"published":5,"total":5},"queue":{"acked":5}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.26,"15":0.15,"5":0.15,"norm":{"1":0.13,"15":0.075,"5":0.075}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:09:08.902Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":4096}}}},"cpu":{"system":{"ticks":980,"time":{"ms":30}},"total":{"ticks":3790,"time":{"ms":90},"value":0},"user":{"ticks":2810,"time":{"ms":60}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":273218},"version":"8.3.2"},"memstats":{"gc_next":22082824,"memory_alloc":10828096,"memory_total":332579344,"rss":107094016},"runtime":{"goroutines":107}},"filebeat":{"events":{"active":1,"added":7,"done":6},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":6,"active":0,"batches":5,"total":6},"read":{"bytes":2075},"write":{"bytes":9206}},"pipeline":{"clients":26,"events":{"active":1,"published":7,"total":7},"queue":{"acked":6}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.22,"15":0.15,"5":0.15,"norm":{"1":0.11,"15":0.075,"5":0.075}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:09:38.903Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":131072}}}},"cpu":{"system":{"ticks":1030,"time":{"ms":50}},"total":{"ticks":3880,"time":{"ms":90},"value":0},"user":{"ticks":2850,"time":{"ms":40}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":303211},"version":"8.3.2"},"memstats":{"gc_next":22082824,"memory_alloc":11746824,"memory_total":333498072,"rss":107094016},"runtime":{"goroutines":107}},"filebeat":{"events":{"added":5,"done":5},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":5,"active":0,"batches":5,"total":5},"read":{"bytes":1865},"write":{"bytes":7755}},"pipeline":{"clients":26,"events":{"active":1,"published":5,"total":5},"queue":{"acked":5}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.19,"15":0.15,"5":0.15,"norm":{"1":0.095,"15":0.075,"5":0.075}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:10:08.898Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":-131072}}}},"cpu":{"system":{"ticks":1080,"time":{"ms":50}},"total":{"ticks":3970,"time":{"ms":90},"value":0},"user":{"ticks":2890,"time":{"ms":40}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":333207},"version":"8.3.2"},"memstats":{"gc_next":22082824,"memory_alloc":12682696,"memory_total":334433944,"rss":107094016},"runtime":{"goroutines":107}},"filebeat":{"events":{"active":-1,"added":4,"done":5},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":5,"active":0,"batches":5,"total":5},"read":{"bytes":1863},"write":{"bytes":7755}},"pipeline":{"clients":26,"events":{"active":0,"published":4,"total":4},"queue":{"acked":5}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.37,"15":0.17,"5":0.2,"norm":{"1":0.185,"15":0.085,"5":0.1}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-07-31T18:10:38.896Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":4096}}}},"cpu":{"system":{"ticks":1130,"time":{"ms":50}},"total":{"ticks":4040,"time":{"ms":70},"value":0},"user":{"ticks":2910,"time":{"ms":20}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":37},"info":{"ephemeral_id":"c362e1cd-5ba5-4e23-9ef7-3f589f1e6abd","uptime":{"ms":363208},"version":"8.3.2"},"memstats":{"gc_next":22082824,"memory_alloc":13638216,"memory_total":335389464,"rss":107094016},"runtime":{"goroutines":107}},"filebeat":{"events":{"active":1,"added":7,"done":6},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":6,"active":0,"batches":3,"total":6},"read":{"bytes":1765},"write":{"bytes":8217}},"pipeline":{"clients":26,"events":{"active":1,"published":7,"total":7},"queue":{"acked":6}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.22,"15":0.16,"5":0.18,"norm":{"1":0.11,"15":0.08,"5":0.09}}}},"ecs.version":"1.6.0"}}
^C{"log.level":"info","@timestamp":"2022-07-31T18:11:03.772Z","log.origin":{"file.name":"beater/filebeat.go","file.line":425},"message":"Stopping filebeat","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.777Z","log.origin":{"file.name":"beater/crawler.go","file.line":155},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.779Z","log.origin":{"file.name":"beater/crawler.go","file.line":165},"message":"Stopping 1 inputs","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.782Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":190},"message":"Dynamic config reloader stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.786Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":130},"message":"Stopping 1 runners ...","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.786Z","log.origin":{"file.name":"input/input.go","file.line":134},"message":"input ticker stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.786Z","log.origin":{"file.name":"udp/input.go","file.line":114},"message":"Stopping UDP input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.787Z","log.logger":"udp","log.origin":{"file.name":"dgram/handler.go","file.line":73},"message":"Connection has been closed","service.name":"filebeat","address":"192.168.37.2:9004","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.784Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":170},"message":"Stopping input: 11337388005444501392","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.787Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":132},"message":"Input 'filestream' stopped","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.787Z","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.787Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":132},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.788Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":166},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.788Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":137},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.788Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::794967-64768","path":"/var/log/ubuntu-advantage-timer.log","state-id":"native::794967-64768","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.790Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::798116-64768","path":"/var/log/cloud-init.log","state-id":"native::798116-64768","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-31T18:11:03.790Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::798004-64768","path":"/var/log/bootstrap.log","state-id":"native::798004-64768","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-
23

yea i see this line in the logs see @stephenb 07-31T18:04:49.027Z","log.logger":"UDP","log.origin":{"file.name":"dgram/server.go","file.line":99},"message":"Started listening for UDP connection","service.name":"filebeat","ecs.version":"1.6.0"}

24

leave it running for a while and check if you are seeing logs in Discover... why are you killing it so fast..

so did you let it run and then look in Discover to see if you are getting any logs...

You also still have the filestream enabled... (ok but that is making it more difficult to debug) which also makes me think you are You said you turned that off so I do not really know what you are doing.

the filebeat -e command line is just running the same filebeat that will run with systemctl...

I really can not do every step for you ...

I think you can debug from here... keep at it.

25

it still nothing data of fortinet in kibana just i think filebeat collect the data of the server not fortinet

26

Well that is because you did not disable the filestream input which you said you did AND that is making it harder to debug!!! I can only try to help... if you don't do what I ask, I can not help...

Plus you still have
var.syslog_host: 192.168.37.2
instead of
var.syslog_host: 0.0.0.0

BUT there is an easy way to test.

In one terminal
start filebeat and let it run! check to see the UDP message is in the filebeat logs.
$ filebeat -e

In another terminal on the filebeat server

This will send message to the UDP port 9004

$  nc -u 0.0.0.0 9004
Hello World
Bye World
Ack
^C

Then go check discover and put this in the KQL
Bar

event.dataset : "fortinet.firewall"

and you should see the messages... If so that means

  1. Filebeat is working correct and listening on UDP 9004

  2. your FW is not sending the logs to filbeat host or there is connectivity issue

Screen Shot 2022-07-31 at 11.31.51 AM
Screen Shot 2022-07-31 at 11.31.51 AM3318×1594 441 KB

Good Luck!

27

Well that is because **you did not disable the filestream input** which you said you did AND that is making it harder to debug!!! I can only try to help... if you don't do what I ask, I can not help.
how to disable the filestream input ?

28

Plus you still have
var.syslog_host: 192.168.37.2
instead of
var.syslog_host: 0.0.0.0 done

BUT there is an easy way to test.
In one terminal
start filebeat and let it run! check to see the UDP message is in the filebeat logs.
$ filebeat -e
when i try to do this cmd this error appears and i did what you told me here but the error still

You have to stop the filebeat service first 2 can not run at the same time...

systemctl stop filebeat

filebeat -e

root@elkfiras:/home/elkfiras# filebeat -e
{"log.level":"info","@timestamp":"2022-08-01T13:11:01.330Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T13:11:01.330Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: 18f18c9b-6051-49ef-a4ce-e9cce3299b83","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-01T13:11:04.336Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T13:11:04.339Z","log.origin":{"file.name":"instance/beat.go","file.line":391},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-08-01T13:11:04.340Z","log.origin":{"file.name":"instance/beat.go","file.line":1051},"message":"Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
29

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.