filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: false
paths:
- /var/log/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
# Kibana Host
host: "https://192.168.1.13:5601"
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["https://localhost:9200"]
#Elasticsearch Output
# Authentication credentials - either API key or username/password.
username: "elastic"
password: "*****"
ssl:
enabled: true
ca_trusted_fingerprint: "*********"
when checking the configuration everything is ok
syslog:~$ sudo filebeat test config
Config OK
syslog:~$ sudo filebeat test output
elasticsearch: https://localhost:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 127.0.0.1
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 8.8.0
port and connection are ok
detect@syslog:~$ sudo tcpdump -i any port 2055 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
13:47:42.625871 IP 0.0.0.0.2055 > 192.168.1.13.2055: UDP, length 1412
13:47:42.625942 IP 0.0.0.0.2055 > 192.168.1.13.2055: UDP, length 1412
13:47:42.625942 IP 0.0.0.0.2055 > 192.168.1.13.2055: UDP, length 360
I get an error when checking module status
I see this in the service filebeat log
Jun 10 13:47:30 syslog filebeat[942]: {"log.level":"info","@timestamp":"2023-06-10T13:47:30.781Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":22664042}},"memory":{"mem":{"usage":{"bytes":67346432}}}},"cpu":{"system":{"ticks":830},"total":{"ticks"
no data displayed on Dashboard panel: Dashboard Navigation [Filebeat Netflow]
Dashboard Navigation [Filebeat Netflow]
What is the output of the following, apologies I updated the command.
GET _cat/indices/?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open metrics-endpoint.metadata_current_default nEfa9lCtQeuvfaIxhNCOAA 1 0 0 0 247b 247b
green open .internal.alerts-observability.metrics.alerts-default-000001 3JlyeNEyRJuQJOIuscBFDg 1 0 0 0 247b 247b
green open .internal.alerts-observability.logs.alerts-default-000001 fT-8yN6CRnSLjFzeXXD-Yw 1 0 0 0 247b 247b
green open .internal.alerts-observability.uptime.alerts-default-000001 puk6nI1QTwqhV6DstEPAWA 1 0 0 0 247b 247b
green open .fleet-files-endpoint-000001 1c8AFuB1Qkih_E3E0sZs0Q 1 0 0 0 247b 247b
green open .fleet-file-data-agent-000001 YndV7an4R_yBN4FWeOh0mw 1 0 0 0 247b 247b
green open .fleet-files-agent-000001 XcamZWGgRT-zDSpgdGo3kw 1 0 0 0 247b 247b
green open .fleet-file-data-endpoint-000001 bKl9IV36QJ-FbJ1Gd4DPDQ 1 0 0 0 247b 247b
green open .internal.alerts-security.alerts-default-000001 MnxZZD3HQXyM6CBzyzcWwQ 1 0 0 0 247b 247b
green open .internal.alerts-observability.slo.alerts-default-000001 AsyjPnvBQ_Wj0zGgtJGSUw 1 0 0 0 247b 247b
yellow open .ds-filebeat-8.8.0-2023.06.03-000001 vJO3sxa1SR2htgr4MDIC0A 1 1 1278791 0 412.2mb 412.2mb
green open .internal.alerts-observability.apm.alerts-default-000001 CQJnSAmUQg2CAJrKbY1FJw 1 0 0 0 247b 247b
detect@syslog:~$ sudo filebeat -e -d "*"
{"log.level":"info","@timestamp":"2023-06-15T06:24:10.662Z","log.origin":{"file.name":"instance/beat.go","file.line":779},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:10.663Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":870},"message":"Beat metadata path: /var/lib/filebeat/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-15T06:24:10.691Z","log.origin":{"file.name":"instance/beat.go","file.line":787},"message":"Beat ID: 4762b2c2-2127-44f1-9e4f-c0acd8bb1345","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:10.692Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition contains: map[]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:10.692Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition !contains: map[]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-06-15T06:24:13.693Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:13.754Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":130},"message":"add_cloud_metadata: starting to fetch metadata, timeout=3s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:13.778Z","log.logger":"docker","log.origin":{"file.name":"docker/client.go","file.line":49},"message":"Docker client will negotiate the API version on the first request.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:13.804Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"add_docker_metadata/add_docker_metadata.go","file.line":90},"message":"add_docker_metadata: docker environment not detected: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:13.848Z","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":148},"message":"Could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable","service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:16.757Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":174},"message":"add_cloud_metadata: timed-out waiting for all responses","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:16.757Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":133},"message":"add_cloud_metadata: fetchMetadata ran for 3.00266599s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-15T06:24:16.757Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:16.789Z","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], condition=!contains: map[], add_cloud_metadata={}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.parent.pid]], add_kubernetes_metadata","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:16.789Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 4 times","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:17.190Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 3 times","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:17.590Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 2 times","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-15T06:24:17.991Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 1 times","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-15T06:24:18.391Z","log.origin":{"file.name":"instance/beat.go","file.line":426},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-06-15T06:24:18.391Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)
commented out the output.elasticsearch section and included filebeat -e output.console and run filebeat in the foreground and see if you get any network stream.
detect@syslog:~$ sudo filebeat -e
{"log.level":"info","@timestamp":"2023-06-15T06:32:53.499Z","log.origin":{"file.name":"instance/beat.go","file.line":779},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-15T06:32:53.499Z","log.origin":{"file.name":"instance/beat.go","file.line":787},"message":"Beat ID: 4762b2c2-2127-44f1-9e4f-c0acd8bb1345","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-06-15T06:32:56.502Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-15T06:32:56.502Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-15T06:32:56.502Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1299},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"4762b2c2-2127-44f1-9e4f-c0acd8bb1345"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-06-15T06:32:56.502Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1308},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"ae3e3f9194a937d20197a7be5d3cbbacaceeb9cc","libbeat":"8.8.0","time":"2023-05-23T01:46:08.000Z","version":"8.8.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1311},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.19.9"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1317},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-06-15T06:26:08Z","containerized":false,"name":"syslog","ip":["127.0.0.1","::1","192.168.10.63","fe80::20c:29ff:fea6:6eb9","fe80::ecee:eeff:feee:eeee","10.1.131.128","fe80::64e0:54ff:fe3b:a22c"],"kernel_version":"5.4.0-150-generic","mac":["00:0c:29:a6:6e:b9","ee:ee:ee:ee:ee:ee","66:e0:54:3b:a2:2c"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","major":20,"minor":4,"patch":6,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0,"id":"053d81efd282406a9b6c3c51d7f2e141"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1346},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null},"cwd":"/home/detect","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":7015,"ppid":7014,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2023-06-15T06:32:52.900Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.origin":{"file.name":"instance/beat.go","file.line":330},"message":"Setup Beat: filebeat; Version: 8.8.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.origin":{"file.name":"instance/beat.go","file.line":365},"message":"no outputs are defined, please define one under the output section","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.origin":{"file.name":"instance/beat.go","file.line":472},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-06-15T06:32:56.503Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: no outputs are defined, please define one under the output section","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: no outputs are defined, please define one under the output section
This means there is data in the filebeat-8.8.0 datastream so go to
Kibana -> Discover
Select the Data view filebeat-*
Select a Proper Time Range in the Time Picker
And Take a look at the event.dataset field click on it and see if you have any netflow data take a screen shot. Mine is syslog but same concept
If there is netflow click on the lite + sign and if so take a look at the data.
detect@syslog:~$ sudo filebeat -e -d "*"
...
Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)
This means you already have a filebeat running somewhere and you can only have 1 filebeat process running , I suspect that you have it still running as a service.
To run filebeat in the foreground you will need to stop the service first.
sudo systemctl stop filebeat
Then run filebeat -e -d "*"
hi, thanks a lot for your help.
I look in discover and see the data, the fact is that I turned on the syslog module to check the operation of filebeat, most likely this is the data of the work of syslog, and not NetFlow
detect@syslog:~$ sudo docker run -it --rm networkstatic/nflow-generator -t 192.168.10.63 -p 2055
INFO[0000] sending netflow data to a collector ip: 192.168.10.63 and port: 2055.
Use ctrl^c to terminate the app.
FATA[0000] Error connecting to the target collector: write udp 172.17.0.2:37410->192.168.10.63:2055: write: connection refused
{"log.level":"info","@timestamp":"2023-06-24T13:55:41.900Z","log.origin":{"file.name":"instance/beat.go","file.line":779},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:41.900Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":870},"message":"Beat metadata path: /var/lib/filebeat/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-24T13:55:41.924Z","log.origin":{"file.name":"instance/beat.go","file.line":787},"message":"Beat ID: 4762b2c2-2127-44f1-9e4f-c0acd8bb1345","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:41.951Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition contains: map[]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:41.951Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition !contains: map[]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-06-24T13:55:44.952Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:44.987Z","log.logger":"docker","log.origin":{"file.name":"docker/client.go","file.line":49},"message":"Docker client will negotiate the API version on the first request.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:45.014Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":130},"message":"add_cloud_metadata: starting to fetch metadata, timeout=3s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:46.952Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"add_docker_metadata/add_docker_metadata.go","file.line":93},"message":"add_docker_metadata: docker environment detected","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:46.952Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":213},"message":"Start docker containers scanner","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:46.952Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":375},"message":"List containers","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:47.005Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":266},"message":"Fetching events since 2023-06-24 13:55:47.005683912 +0000 UTC m=+5.312141392","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:47.071Z","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":148},"message":"Could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable","service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:48.063Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":174},"message":"add_cloud_metadata: timed-out waiting for all responses","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:48.063Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":133},"message":"add_cloud_metadata: fetchMetadata ran for 3.000541484s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-24T13:55:48.063Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:48.063Z","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], condition=!contains: map[], add_cloud_metadata={}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.parent.pid]], add_kubernetes_metadata","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:48.063Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 4 times","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:48.480Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 3 times","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:48.881Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 2 times","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:49.281Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 1 times","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-06-24T13:55:49.682Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":313},"message":"Watcher stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-24T13:55:49.682Z","log.origin":{"file.name":"instance/beat.go","file.line":426},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-06-24T13:55:49.682Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)
You have to stop the other filebeat before starting filebeat in the foreground with filebeat -e -d "*"
You can only run 1 filebeat at a time... read the error messages they are pretty clear.
Please try again...
I know nothing about mikrotik so I can not help there
var.max_message_size
The maximum size of the message received over UDP. The default is 10KiB.
I see 32K in your Mikrotik settings perhaps increase the max message size large in the netflow.yml to 32K or larger or decrease the size in the Mikrotik.... but that says cache so that may have nothing to do with it...
Perhaps try a different format like version 9.
Or perhaps look at some of the other setting in the filebeat netflow module
var.read_buffer
The size of the read buffer on the UDP socket. var.timeout
The read and write timeout for socket operations. var.expiration_timeout
The time before an idle session or unused template is expired. Only applicable to v9 and IPFIX protocols. A value of zero disables expiration. var.queue_size
The maximum number of packets that can be queued for processing. Use this setting to avoid packet-loss when dealing with occasional bursts of traffic.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
