Hello,
I've installed the Fortinet Logs Integration and configured the Fortigate to send syslogs for 9004. I was able to test that the Fortigate is sending logs but nothing shows up in Elastic. Is there something I am missing? It doesn't look like anything is listening on the 9004 port either. I'm kinda stuck here, not sure where to go now. Any help would be greatly appreciated.
Hi @jumpie Welcome to the community.,, we are going to need a little more information...
How did you install filebeat?
Did you follow all these steps exactly but with the fortinet module?
Please post your filebeat.yml and the fortinet.yml
How are you starting filebeat?
Did you look at the filebeat logs? Are there any errors?
Thanks for replying @stephenb !
I checked the logs for Filebeat and don't see much. I did follow the steps you linked and filebeat seems to be running. Below is the requested configs and the log.
Log:
"log.level":"info","@timestamp":"2022-05-03T14:49:47.537Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/home/bottlebill/filebeat-8.2.0-linux-x86_64] Config path: [/home/bottlebill/filebeat-8.2.0-linux-x86_64] Data path: [/home/bottlebill/filebeat-8.2.0-linux-x86_64/data] Logs path: [/home/bottlebill/filebeat-8.2.0-linux-x86_64/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T14:49:47.537Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: d309f519-a6ba-4880-9037-e13269ca7dd7","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-03T14:49:50.540Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
filebeat.yml
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: false
paths:
- /var/log/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
host: "10.10.10.10:5601"
username: "user"
password: "password"
output.elasticsearch:
hosts: ["https://localhost:9200"]
ssl.verification_mode: none
username: "user"
password: "password"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
fortinet.yml
- module: fortinet
firewall:
enabled: true
clientendpoint:
enabled: false
fortimail:
enabled: false
fortimanager:
enabled: false
In the future please format your code... using the </> button above (i did that because I could not read it well 
)
What version are you on? 8.2?
Not sure "what don't see much means"
Please try to be a bit more specific... like
I went to discover in Kibana and I did not see any logs under the filebeat-* index pattern... is that what you mean?
Make sure your forinet is sending to the correct ip address and port? and have the correct interface.
So perhaps set the setting that will work with your expected config... example
- module: fortinet
firewall:
enabled: true
var.input: udp
var.syslog_host: 0.0.0.0
var.syslog_port: 9004
You should also see a log lines like the following in the filebeat startup... there are many logs on startup 50 or so lines..
{"log.level":"info","@timestamp":"2022-05-03T09:51:08.022-0700","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":133},"message":"Elasticsearch pipeline loaded.","service.name":"filebeat","pipeline":"filebeat-8.1.2-fortinet-firewall-pipeline","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:51:08.121-0700","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":133},"message":"Elasticsearch pipeline loaded.","service.name":"filebeat","pipeline":"filebeat-8.1.2-fortinet-firewall-event","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:51:08.213-0700","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":133},"message":"Elasticsearch pipeline loaded.","service.name":"filebeat","pipeline":"filebeat-8.1.2-fortinet-firewall-utm","ecs.version":"1.6.0"}
....
{"log.level":"info","@timestamp":"2022-05-03T09:51:08.297-0700","log.origin":{"file.name":"udp/input.go","file.line":99},"message":"Starting UDP input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:51:08.297-0700","log.logger":"UDP","log.origin":{"file.name":"dgram/server.go","file.line":99},"message":"Started listening for UDP connection","service.name":"filebeat","ecs.version":"1.6.0"}
Okay, sorry. I'll use the </> from now on. I'm using Filebeat 8.2.0. The comment in fortinet.yml above the syslog_port sayd it defaults to 9004. I did add the fields like in the example to the fortinet.yml. Not seeing any syslogs coming through though and the nothing is listening on port 9004.
Filebeat log from start:
user@es01:~/filebeat-8.2.0-linux-x86_64$ sudo ./filebeat -e
{"log.level":"info","@timestamp":"2022-05-03T09:52:26.216-0700","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/home/user/filebeat-8.2.0-linux-x86_64] Config path: [/home/user/filebeat-8.2.0-linux-x86_64] Data path: [/home/user/filebeat-8.2.0-linux-x86_64/data] Logs path: [/home/user/filebeat-8.2.0-linux-x86_64/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:26.216-0700","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: d309f519-a6ba-4880-9037-e13269ca7dd7","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-03T09:52:29.219-0700","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.221-0700","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.221-0700","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/home/user/filebeat-8.2.0-linux-x86_64","data":"/home/user/filebeat-8.2.0-linux-x86_64/data","home":"/home/user/filebeat-8.2.0-linux-x86_64","logs":"/home/user/filebeat-8.2.0-linux-x86_64/logs"},"type":"filebeat","uuid":"d309f519-a6ba-4880-9037-e13269ca7dd7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.221-0700","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1072},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"045da3a1bb89944373c33332c18ca99ef6192df2","libbeat":"8.2.0","time":"2022-04-19T23:31:06.000Z","version":"8.2.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.221-0700","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1075},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":6,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.221-0700","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-02T14:25:09-07:00","containerized":false,"name":"es01","ip":["127.0.0.1/8","::1/128","10.20.30.115/24","fe80::215:5dff:fe04:d24/64"],"kernel_version":"5.4.0-109-generic","mac":["00:15:5d:04:0d:24"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"PDT","timezone_offset_sec":-25200,"id":"721ec161f76e4baf9de749a6e09b391d"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.222-0700","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null},"cwd":"/home/user/filebeat-8.2.0-linux-x86_64","exe":"/home/user/filebeat-8.2.0-linux-x86_64/filebeat","name":"filebeat","pid":9921,"ppid":9920,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-05-03T09:52:25.470-0700"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.222-0700","log.origin":{"file.name":"instance/beat.go","file.line":325},"message":"Setup Beat: filebeat; Version: 8.2.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-03T09:52:29.223-0700","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.223-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: https://localhost:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-03T09:52:29.223-0700","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.223-0700","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: es01","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.223-0700","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.223-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.223-0700","log.origin":{"file.name":"instance/beat.go","file.line":505},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.224-0700","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/home/user/filebeat-8.2.0-linux-x86_64/data/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.224-0700","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.224-0700","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.224-0700","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.224-0700","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":121},"message":"input disabled, skipping it","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.226-0700","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: fortinet (firewall)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.226-0700","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.226-0700","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.227-0700","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: fortinet (firewall)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.228-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: https://localhost:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-03T09:52:29.228-0700","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-03T09:52:29.228-0700","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.256-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.265-0700","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.265-0700","log.origin":{"file.name":"udp/input.go","file.line":98},"message":"Starting UDP input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:29.265-0700","log.logger":"UDP","log.origin":{"file.name":"dgram/server.go","file.line":99},"message":"Started listening for UDP connection","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:32.221-0700","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":101},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:37.747-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(https://localhost:9200))","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-03T09:52:37.748-0700","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:37.775-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:37.781-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-03T09:52:39.574-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3, Beat=8.2.0.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:39.575-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(https://localhost:9200)) with 1 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:39.576-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-03T09:52:41.990-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3, Beat=8.2.0.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:41.990-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(https://localhost:9200)) with 2 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:41.991-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-03T09:52:46.800-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3, Beat=8.2.0.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:46.800-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(https://localhost:9200)) with 3 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:46.801-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-03T09:52:58.582-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3, Beat=8.2.0.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:58.582-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(https://localhost:9200)) with 4 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:58.583-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:52:59.227-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"user.slice"},"cpuacct":{"id":"user.slice","total":{"ns":8308597454590}},"memory":{"id":"session-8.scope","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":702971904}}}},"cpu":{"system":{"ticks":280,"time":{"ms":280}},"total":{"ticks":840,"time":{"ms":840},"value":0},"user":{"ticks":560,"time":{"ms":560}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":13},"info":{"ephemeral_id":"14c8ce41-c547-46b8-8cf1-e6178435730b","uptime":{"ms":33087},"version":"8.2.0"},"memstats":{"gc_next":56125168,"memory_alloc":33099736,"memory_sys":59327496,"memory_total":145374544,"rss":153948160},"runtime":{"goroutines":36}},"filebeat":{"events":{"active":4104,"added":4104},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":1},"output":{"events":{"active":0},"read":{"bytes":5702},"type":"elasticsearch","write":{"bytes":3050}},"pipeline":{"clients":1,"events":{"active":4100,"published":4100,"retry":2004,"total":4100},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":6},"load":{"1":0.09,"15":0.14,"5":0.2,"norm":{"1":0.015,"15":0.0233,"5":0.0333}}}},"ecs.version":"1.6.0"}}
{"log.level":"error","@timestamp":"2022-05-03T09:53:15.219-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3, Beat=8.2.0.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:53:15.220-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(https://localhost:9200)) with 5 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:53:15.221-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:53:29.226-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":3171815859}},"memory":{"mem":{"usage":{"bytes":-98304}}}},"cpu":{"system":{"ticks":280},"total":{"ticks":850,"time":{"ms":10},"value":0},"user":{"ticks":570,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":13},"info":{"ephemeral_id":"14c8ce41-c547-46b8-8cf1-e6178435730b","uptime":{"ms":63086},"version":"8.2.0"},"memstats":{"gc_next":56125168,"memory_alloc":33815080,"memory_total":146089888,"rss":153948160},"runtime":{"goroutines":36}},"filebeat":{"events":{"active":13,"added":13},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0},"read":{"bytes":633},"write":{"bytes":297}},"pipeline":{"clients":1,"events":{"active":4117,"published":16,"retry":501,"total":17}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.06,"15":0.14,"5":0.18,"norm":{"1":0.01,"15":0.0233,"5":0.03}}}},"ecs.version":"1.6.0"}}
{"log.level":"error","@timestamp":"2022-05-03T09:53:48.692-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3, Beat=8.2.0.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:53:48.692-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(https://localhost:9200)) with 6 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:53:48.693-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:53:59.226-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":6139397272}},"memory":{"mem":{"usage":{"bytes":-102400}}}},"cpu":{"system":{"ticks":290,"time":{"ms":10}},"total":{"ticks":860,"time":{"ms":10},"value":0},"user":{"ticks":570}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":12},"info":{"ephemeral_id":"14c8ce41-c547-46b8-8cf1-e6178435730b","uptime":{"ms":93086},"version":"8.2.0"},"memstats":{"gc_next":56125168,"memory_alloc":34156192,"memory_total":146431000,"rss":153948160},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0},"read":{"bytes":1109},"write":{"bytes":614}},"pipeline":{"clients":1,"events":{"active":4117,"retry":501}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.03,"15":0.13,"5":0.16,"norm":{"1":0.005,"15":0.0217,"5":0.0267}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-03T09:54:29.226-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":3528524654}},"memory":{"mem":{"usage":{"bytes":139264}}}},"cpu":{"system":{"ticks":290},"total":{"ticks":870,"time":{"ms":10},"value":0},"user":{"ticks":580,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":12},"info":{"ephemeral_id":"14c8ce41-c547-46b8-8cf1-e6178435730b","uptime":{"ms":123086},"version":"8.2.0"},"memstats":{"gc_next":56125168,"memory_alloc":34466024,"memory_total":146740832,"rss":153948160},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":4117}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.02,"15":0.12,"5":0.15,"norm":{"1":0.0033,"15":0.02,"5":0.025}}}},"ecs.version":"1.6.0"}}
{"log.level":"error","@timestamp":"2022-05-03T09:54:31.690-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3, Beat=8.2.0.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:54:31.690-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(https://localhost:9200)) with 7 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:54:31.692-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:54:59.229-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":3471328721}},"memory":{"mem":{"usage":{"bytes":-122880}}}},"cpu":{"system":{"ticks":290},"total":{"ticks":900,"time":{"ms":30},"value":0},"user":{"ticks":610,"time":{"ms":30}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":12},"info":{"ephemeral_id":"14c8ce41-c547-46b8-8cf1-e6178435730b","uptime":{"ms":153088},"version":"8.2.0"},"memstats":{"gc_next":58292848,"memory_alloc":29328592,"memory_total":147061912,"rss":153948160},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0},"read":{"bytes":1109},"write":{"bytes":614}},"pipeline":{"clients":1,"events":{"active":4117,"retry":501}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.01,"15":0.12,"5":0.13,"norm":{"1":0.0017,"15":0.02,"5":0.0217}}}},"ecs.version":"1.6.0"}}
{"log.level":"error","@timestamp":"2022-05-03T09:55:04.413-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3, Beat=8.2.0.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:55:04.414-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(https://localhost:9200)) with 8 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:55:04.414-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:55:29.226-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":3315618088}},"memory":{"mem":{"usage":{"bytes":-1007616}}}},"cpu":{"system":{"ticks":310,"time":{"ms":20}},"total":{"ticks":940,"time":{"ms":40},"value":0},"user":{"ticks":630,"time":{"ms":20}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":12},"info":{"ephemeral_id":"14c8ce41-c547-46b8-8cf1-e6178435730b","uptime":{"ms":183086},"version":"8.2.0"},"memstats":{"gc_next":58292848,"memory_alloc":29490904,"memory_total":147224224,"rss":153059328},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0},"read":{"bytes":1109},"write":{"bytes":614}},"pipeline":{"clients":1,"events":{"active":4117,"retry":501}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.11,"5":0.12,"norm":{"1":0,"15":0.0183,"5":0.02}}}},"ecs.version":"1.6.0"}}
{"log.level":"error","@timestamp":"2022-05-03T09:55:49.678-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3, Beat=8.2.0.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:55:49.678-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(https://localhost:9200)) with 9 reconnect attempt(s)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:55:49.679-0700","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":287},"message":"Attempting to connect to Elasticsearch version 8.1.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-03T09:55:59.226-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":3117305294}},"memory":{"mem":{"usage":{"bytes":32768}}}},"cpu":{"system":{"ticks":320,"time":{"ms":10}},"total":{"ticks":960,"time":{"ms":20},"value":0},"user":{"ticks":640,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":12},"info":{"ephemeral_id":"14c8ce41-c547-46b8-8cf1-e6178435730b","uptime":{"ms":213086},"version":"8.2.0"},"memstats":{"gc_next":58292848,"memory_alloc":30027680,"memory_sys":262144,"memory_total":147761000,"rss":153059328},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0},"read":{"bytes":1109},"write":{"bytes":614}},"pipeline":{"clients":1,"events":{"active":4117,"retry":501}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.11,"5":0.1,"norm":{"1":0,"15":0.0183,"5":0.0167}}}},"ecs.version":"1.6.0"}}
Good the filebeat module started... and should be listening...
BAD... filebeat can not connect to elastic... what version of elastic are you using? easier to probably use the same filebeat version as elastic...
You should probably match it with the filebeat version.
"message":"Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.1.3,
I have not tried the
output.elasticsearch.allow_older_versions: true
So I can not comment on how that works
Running Elasticsearch 8.1.3. I ran apt update && apt upgrade but it only upgraded Logstash. I can seem to figure out how to update/upgrade Elasticsearch
Why not just download filebeat 8.1.3 and try.
I think 8.2 Elasticsearch is just rolling out today... perhaps in a couple hours it will be available..
Or try the setting... in the filebeat.yml
output.elasticsearch.allow_older_versions: true
Its working! I installed Filebeat 8.1.3
Thank you for all you help Stephen! 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.