# Netflow Mikrotik no data in elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [June 10, 2023, 1:59pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692 "2023-06-10T13:59:40Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 10, 2023, 1:59pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/1 "2023-06-10T13:59:40Z")
hello please help, installed elastic 8.8 + kibana filebeat + netflow I don't see data in my Elasticsearch also when checking the netflow module - check data - No data has been received from this module yet /etc/filebeat/modules.d/netflow.yml ```auto - module: netflow log: enabled: true var: netflow_host: 0.0.0.0 netflow_port: 2055 internal_networks: - private ``` /etc/filebeat/filebeat.yml ```auto filebeat.inputs: - type: filestream id: my-filestream-id enabled: false paths: - /var/log/*.log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 1 setup.kibana: # Kibana Host host: "https://192.168.1.13:5601" output.elasticsearch: # Array of hosts to connect to. hosts: ["https://localhost:9200"] #Elasticsearch Output # Authentication credentials - either API key or username/password. username: "elastic" password: " *****" ssl: enabled: true ca_trusted_fingerprint: " *********" ``` when checking the configuration everything is ok ```auto syslog:~$ sudo filebeat test config Config OK syslog:~$ sudo filebeat test output elasticsearch: https://localhost:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 127.0.0.1 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 8.8.0 ``` port and connection are ok ```auto detect@syslog:~$ sudo tcpdump -i any port 2055 -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes 13:47:42.625871 IP 0.0.0.0.2055 > 192.168.1.13.2055: UDP, length 1412 13:47:42.625942 IP 0.0.0.0.2055 > 192.168.1.13.2055: UDP, length 1412 13:47:42.625942 IP 0.0.0.0.2055 > 192.168.1.13.2055: UDP, length 360 ``` I get an error when checking module status ![111](https://us1.discourse-cdn.com/elastic/original/3X/2/a/2a9149e4f2b0ddcc40256c246ce6ea77d122a333.jpeg) I see this in the service filebeat log ```auto Jun 10 13:47:30 syslog filebeat[942]: {"log.level":"info","@timestamp":"2023-06-10T13:47:30.781Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":22664042}},"memory":{"mem":{"usage":{"bytes":67346432}}}},"cpu":{"system":{"ticks":830},"total":{"ticks" ``` no data displayed on Dashboard panel: Dashboard Navigation [Filebeat Netflow] Dashboard Navigation [Filebeat Netflow] ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 12, 2023, 12:39am UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/2 "2023-06-12T00:39:56Z")
Hi @sana1567 Welcome to the community! What method did you use install? Are there any error messages in the log? Unfortunately you truncated that log line so we can not see the event.published etc. Perhaps look again. Couple things come to mind.. Did you run `filebeat setup -e` _ **Before** _ you started filebeat? Did you run that command or some other command? This sets up all the parsing etc Also you could do a couple things to debug. In the filebeat.yml you could comment out the `output.elasticsearch` section and enable the `output.console` and start filebeat in the foreground and see if you are receiving any netfow in. You could also start in the foreground with `filebeat -e -d "*"` Which will show everything getting published. Can you also go into Kibana - Dev Tools and run `GET _cat/indices/filebeat*/?v` Let us know what you see.. ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 13, 2023, 10:55am UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/3 "2023-06-13T10:55:13Z")
Hello, I am very grateful for the answer. I used the installation method from deb packages. Before command execution ```auto filebeat setup -e ``` just installed elastic and kibana Did you run that command or some other command? - ```auto sudo filebeat setup ``` Dev Tools and run ![elastick](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8bbe0028f024194a782e1e50ff5d58c9bda2302d.jpeg) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 13, 2023, 1:54pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/4 "2023-06-13T13:54:20Z")
Hi @sana1567 Good but we need more What is the output of the following, apologies I updated the command. `GET _cat/indices/?v` Please run it and show the results in text not a screenshot please. Also did you try the other suggestions, and look in the logs, test the output. Set the debug flags as I described ... Those will help too ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 15, 2023, 6:35am UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/5 "2023-06-15T06:35:49Z")
What is the output of the following, apologies I updated the command. `GET _cat/indices/?v` ```auto health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open metrics-endpoint.metadata_current_default nEfa9lCtQeuvfaIxhNCOAA 1 0 0 0 247b 247b green open .internal.alerts-observability.metrics.alerts-default-000001 3JlyeNEyRJuQJOIuscBFDg 1 0 0 0 247b 247b green open .internal.alerts-observability.logs.alerts-default-000001 fT-8yN6CRnSLjFzeXXD-Yw 1 0 0 0 247b 247b green open .internal.alerts-observability.uptime.alerts-default-000001 puk6nI1QTwqhV6DstEPAWA 1 0 0 0 247b 247b green open .fleet-files-endpoint-000001 1c8AFuB1Qkih_E3E0sZs0Q 1 0 0 0 247b 247b green open .fleet-file-data-agent-000001 YndV7an4R_yBN4FWeOh0mw 1 0 0 0 247b 247b green open .fleet-files-agent-000001 XcamZWGgRT-zDSpgdGo3kw 1 0 0 0 247b 247b green open .fleet-file-data-endpoint-000001 bKl9IV36QJ-FbJ1Gd4DPDQ 1 0 0 0 247b 247b green open .internal.alerts-security.alerts-default-000001 MnxZZD3HQXyM6CBzyzcWwQ 1 0 0 0 247b 247b green open .internal.alerts-observability.slo.alerts-default-000001 AsyjPnvBQ_Wj0zGgtJGSUw 1 0 0 0 247b 247b yellow open .ds-filebeat-8.8.0-2023.06.03-000001 vJO3sxa1SR2htgr4MDIC0A 1 1 1278791 0 412.2mb 412.2mb green open .internal.alerts-observability.apm.alerts-default-000001 CQJnSAmUQg2CAJrKbY1FJw 1 0 0 0 247b 247b ``` ```auto detect@syslog:~$ sudo filebeat -e -d "*" {"log.level":"info","@timestamp":"2023-06-15T06:24:10.662Z","log.origin":{"file.name":"instance/beat.go","file.line":779},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:10.663Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":870},"message":"Beat metadata path: /var/lib/filebeat/meta.json","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-15T06:24:10.691Z","log.origin":{"file.name":"instance/beat.go","file.line":787},"message":"Beat ID: 4762b2c2-2127-44f1-9e4f-c0acd8bb1345","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:10.692Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition contains: map[]","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:10.692Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition !contains: map[]","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-06-15T06:24:13.693Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:13.754Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":130},"message":"add_cloud_metadata: starting to fetch metadata, timeout=3s","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:13.778Z","log.logger":"docker","log.origin":{"file.name":"docker/client.go","file.line":49},"message":"Docker client will negotiate the API version on the first request.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:13.804Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"add_docker_metadata/add_docker_metadata.go","file.line":90},"message":"add_docker_metadata: docker environment not detected: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:13.848Z","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":148},"message":"Could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable","service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:16.757Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":174},"message":"add_cloud_metadata: timed-out waiting for all responses","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:16.757Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":133},"message":"add_cloud_metadata: fetchMetadata ran for 3.00266599s","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-15T06:24:16.757Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:16.789Z","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], condition=!contains: map[], add_cloud_metadata={}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.parent.pid]], add_kubernetes_metadata","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:16.789Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 4 times","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:17.190Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 3 times","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:17.590Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 2 times","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-15T06:24:17.991Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 1 times","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-15T06:24:18.391Z","log.origin":{"file.name":"instance/beat.go","file.line":426},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2023-06-15T06:24:18.391Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)","service.name":"filebeat","ecs.version":"1.6.0"} Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data) ``` commented out the output.elasticsearch section and included `filebeat -e` output.console and run filebeat in the foreground and see if you get any network stream. ```auto detect@syslog:~$ sudo filebeat -e {"log.level":"info","@timestamp":"2023-06-15T06:32:53.499Z","log.origin":{"file.name":"instance/beat.go","file.line":779},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-15T06:32:53.499Z","log.origin":{"file.name":"instance/beat.go","file.line":787},"message":"Beat ID: 4762b2c2-2127-44f1-9e4f-c0acd8bb1345","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-06-15T06:32:56.502Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-15T06:32:56.502Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-15T06:32:56.502Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1299},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"4762b2c2-2127-44f1-9e4f-c0acd8bb1345"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-06-15T06:32:56.502Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1308},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"ae3e3f9194a937d20197a7be5d3cbbacaceeb9cc","libbeat":"8.8.0","time":"2023-05-23T01:46:08.000Z","version":"8.8.0"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1311},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.19.9"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1317},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-06-15T06:26:08Z","containerized":false,"name":"syslog","ip":["127.0.0.1","::1","192.168.10.63","fe80::20c:29ff:fea6:6eb9","fe80::ecee:eeff:feee:eeee","10.1.131.128","fe80::64e0:54ff:fe3b:a22c"],"kernel_version":"5.4.0-150-generic","mac":["00:0c:29:a6:6e:b9","ee:ee:ee:ee:ee:ee","66:e0:54:3b:a2:2c"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","major":20,"minor":4,"patch":6,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0,"id":"053d81efd282406a9b6c3c51d7f2e141"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1346},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null},"cwd":"/home/detect","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":7015,"ppid":7014,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2023-06-15T06:32:52.900Z"},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.origin":{"file.name":"instance/beat.go","file.line":330},"message":"Setup Beat: filebeat; Version: 8.8.0","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.origin":{"file.name":"instance/beat.go","file.line":365},"message":"no outputs are defined, please define one under the output section","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-15T06:32:56.503Z","log.origin":{"file.name":"instance/beat.go","file.line":472},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2023-06-15T06:32:56.503Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: no outputs are defined, please define one under the output section","service.name":"filebeat","ecs.version":"1.6.0"} Exiting: no outputs are defined, please define one under the output section ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 15, 2023, 2:34pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/6 "2023-06-15T14:34:19Z")
Hi @sana1567 Ok Good stuff > [@sana1567](#): > > ```auto > yellow open .ds-filebeat-8.8.0-2023.06.03-000001 vJO3sxa1SR2htgr4MDIC0A 1 1 1278791 0 412.2mb 412.2mb > > ``` This means there is data in the filebeat-8.8.0 datastream so go to Kibana -\> Discover Select the Data view `filebeat-*` Select a Proper Time Range in the Time Picker And Take a look at the `event.dataset` field click on it and see if you have any netflow data take a screen shot. Mine is syslog but same concept If there is netflow click on the lite `+` sign and if so take a look at the data. ![Screenshot 2023-06-15 at 7.26.47 AM](https://us1.discourse-cdn.com/elastic/original/3X/e/9/e97b4e387e80b4361978639c4eb1dfa0f18dad16.jpeg) ```auto detect@syslog:~$ sudo filebeat -e -d "*" ... Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data) ``` This means you already have a filebeat running somewhere and you can only have 1 filebeat process running , I suspect that you have it still running as a service. To run filebeat in the foreground you will need to stop the service first. `sudo systemctl stop filebeat` Then run `filebeat -e -d "*"` > [@sana1567](#): > > `Exiting: no outputs are defined, please define one under the output section` sorry I guess a typo on my part `output.console:` Need the `:` `..............^` ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 16, 2023, 10:57am UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/7 "2023-06-16T10:57:58Z")
hi, thanks a lot for your help. I look in discover and see the data, the fact is that I turned on the syslog module to check the operation of filebeat, most likely this is the data of the work of syslog, and not NetFlow ![event](https://us1.discourse-cdn.com/elastic/original/3X/d/3/d34532e3827c6d348f58bd96aca6fa206e12b255.jpeg) didn't understand about - ```auto sorry I guess a typo on my part `output.console:` Need the `:` `..............^` ``` I already tried this, I found it on google, it did not help `sudo systemctl stop filebeat` Then run `filebeat -e -d "*"` ```auto {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"reader_multiline","log.origin":{"file.name":"multiline/pattern.go","file.line":142},"message":"Multiline event flushed because timeout reached.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":340},"message":"End of file reached. Closing because close_eof is enabled.","service.name":"filebeat","input_id":"bde8b4ce-23b3-448e-86a9-755a029bc438","source_file":"/var/log/auth.log","state_id":"native::1051852-64768","finished":false,"os_id":"1051852-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051852-64768","harvester_id":"364eb1bb-1dad-4c39-9501-0a3005b8d47b","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":616},"message":"Stopping harvester.","service.name":"filebeat","input_id":"bde8b4ce-23b3-448e-86a9-755a029bc438","source_file":"/var/log/auth.log","state_id":"native::1051852-64768","finished":false,"os_id":"1051852-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051852-64768","harvester_id":"364eb1bb-1dad-4c39-9501-0a3005b8d47b","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":626},"message":"Closing file","service.name":"filebeat","input_id":"bde8b4ce-23b3-448e-86a9-755a029bc438","source_file":"/var/log/auth.log","state_id":"native::1051852-64768","finished":false,"os_id":"1051852-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051852-64768","harvester_id":"364eb1bb-1dad-4c39-9501-0a3005b8d47b","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":490},"message":"Update state (offset: 38468).","service.name":"filebeat","input_id":"bde8b4ce-23b3-448e-86a9-755a029bc438","source_file":"/var/log/auth.log","state_id":"native::1051852-64768","finished":false,"os_id":"1051852-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051852-64768","harvester_id":"364eb1bb-1dad-4c39-9501-0a3005b8d47b","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":637},"message":"harvester cleanup finished.","service.name":"filebeat","input_id":"bde8b4ce-23b3-448e-86a9-755a029bc438","source_file":"/var/log/auth.log","state_id":"native::1051852-64768","finished":false,"os_id":"1051852-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051852-64768","harvester_id":"364eb1bb-1dad-4c39-9501-0a3005b8d47b","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":145},"message":"client: closing acker","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":150},"message":"client: done closing acker","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":152},"message":"client: close queue producer","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":176},"message":"client: cancelled 0 events","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":155},"message":"client: done producer close","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":158},"message":"client: closing processors","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":163},"message":"client: done closing processors","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":152},"message":"Stopped runner: system (syslog), system (auth)","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-16T10:54:31.449Z","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-16T10:54:31.449Z","log.origin":{"file.name":"beater/filebeat.go","file.line":460},"message":"Stopping filebeat","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":132},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":166},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":167},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-16T10:54:31.449Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":137},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-16T10:54:31.471Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":195},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"user.slice","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"user.slice","total":{"ns":113338002421}},"memory":{"id":"session-1.scope","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":149659648}}}},"cpu":{"system":{"ticks":90,"time":{"ms":90}},"total":{"ticks":530,"time":{"ms":530},"value":530},"user":{"ticks":440,"time":{"ms":440}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"bf6edfa4-d617-4c3a-bfc3-693cf3994370","name":"filebeat","uptime":{"ms":168580},"version":"8.8.0"},"memstats":{"gc_next":21528992,"memory_alloc":12655408,"memory_sys":38057224,"memory_total":143373720,"rss":97427456},"runtime":{"goroutines":15}},"filebeat":{"events":{"active":1,"added":195,"done":194},"harvester":{"closed":2,"open_files":0,"running":0,"skipped":0,"started":2},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":2,"starts":2,"stops":0},"reloads":1,"scans":1},"output":{"batches":{"split":0},"events":{"acked":189,"active":0,"batches":53,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":189},"read":{"bytes":65459,"errors":0},"type":"elasticsearch","write":{"bytes":252937,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":6,"published":189,"retry":50,"total":195},"queue":{"acked":189,"max_events":4096}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"registrar":{"states":{"cleanup":0,"current":4,"update":195},"writes":{"fail":0,"success":51,"total":51}},"system":{"cpu":{"cores":4},"load":{"1":0.29,"15":0.24,"5":0.32,"norm":{"1":0.0725,"15":0.06,"5":0.08}}}},"ecs.version":"1.6.0"}} ``` ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 16, 2023, 10:59am UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/8 "2023-06-16T10:59:01Z")
if you open syslog, then it works ![event1](https://us1.discourse-cdn.com/elastic/original/3X/3/8/387b34718c1b53549643e5289969d1c20965374e.jpeg) ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 16, 2023, 10:59am UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/9 "2023-06-16T10:59:47Z")
I don't know what to do next, there is still no data here ![event2](https://us1.discourse-cdn.com/elastic/original/3X/5/9/59158aa324dc4a4df760f6a4a2573d92a4f70a06.jpeg) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 16, 2023, 6:34pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/10 "2023-06-16T18:34:44Z")
@sana1567 So it looks to me that you are not actually receiving any netflow data. If you were you would see a lot of output when you ran `filebeat -e -d "*"` So there is a way to check ... I ran a local netflow generator to test this is my netflow.yml ```auto - module: netflow log: enabled: true var: netflow_host: 0.0.0.0 netflow_port: 2055 internal_networks: - private ``` start filebeat with this.. `filebeat -e -d "*"` Then use this generator... > **[GitHub - nerdalert/nflow-generator: NetFlow Generator for Testing Flow...](https://github.com/nerdalert/nflow-generator)** > > NetFlow Generator for Testing Flow Collection Apps - GitHub - nerdalert/nflow-generator: NetFlow Generator for Testing Flow Collection Apps `docker run -it --rm networkstatic/nflow-generator -t 192.168.2.121 -p 2055` Make sure you put in the IP of the server that filebeat is listening on. I ran it on the same server / right next to filebeat... It generated lots of "fake" netflow data... ![Screenshot 2023-06-16 at 11.32.23 AM](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f63d176819c44c54af9a1158c69647404885f384.jpeg) ![Screenshot 2023-06-16 at 11.31.46 AM](https://us1.discourse-cdn.com/elastic/original/3X/c/4/c42d61722ff2496e4920c98276a91536c0971a36.jpeg) If this works then you have a different issue... network connectivity etc. The netflow data is not making it to the filebeat listener. ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 17, 2023, 12:26pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/11 "2023-06-17T12:26:43Z")
some problem with filebeat tried to use ```auto - module: netflow log: enabled: true var: # netflow_host: 0.0.0.0 netflow_host: localhost netflow_port: 2055 internal_networks: - private ``` when running docker ```auto detect@syslog:~$ sudo docker run -it --rm networkstatic/nflow-generator -t 192.168.10.63 -p 2055 INFO[0000] sending netflow data to a collector ip: 192.168.10.63 and port: 2055. Use ctrl^c to terminate the app. FATA[0000] Error connecting to the target collector: write udp 172.17.0.2:37410->192.168.10.63:2055: write: connection refused ``` when checking the host I see ```auto detect@syslog:~$ sudo netstat -tulnp | grep 2055 [sudo] password for detect: udp 0 0 127.0.0.1:2055 0.0.0.0:* 969/filebeat ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 17, 2023, 3:02pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/12 "2023-06-17T15:02:13Z")
> [@sana1567](#): > > `netflow_host: localhost` That won't work leave it `0.0.0.0` for the test. Set that and try the test again ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 24, 2023, 8:48am UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/13 "2023-06-24T08:48:50Z")
hi thanks for the help, fixed the configuration, thanks, the netflow generator worked, but for some reason traffic from my microtik does not come ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 24, 2023, 1:03pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/14 "2023-06-24T13:03:54Z")
tell me why it is not possible to receive traffic, because it comes from Microtick ```auto detect@syslog:~$ sudo tcpdump -i any port 2055 -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes 13:00:50.539874 IP 0.0.0.0.2055 > 192.168.10.63.2055: UDP, length 972 13:00:51.589440 IP 0.0.0.0.2055 > 192.168.10.63.2055: UDP, length 1400 13:00:51.589487 IP 0.0.0.0.2055 > 192.168.10.63.2055: UDP, length 1400 13:00:51.589545 IP 0.0.0.0.2055 > 192.168.10.63.2055: UDP, length 1400 13:00:52.619351 IP 0.0.0.0.2055 > 192.168.10.63.2055: UDP, length 1400 ^C ``` I check the port from another PC - ok ```auto root@ps:/# nc -z -v -u -w3 192.168.10.63 2055 Connection to 192.168.10.63 2055 port [udp/*] succeeded! ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 24, 2023, 1:42pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/15 "2023-06-24T13:42:46Z")
> [@sana1567](#): > > tell me why it is not possible to receive traffic, because it comes from Microtick I have no clue. > [@stephenb](#): > > filebeat -e -d "\*" Did you try starting filebeat with this to see if you see any output? Now that you have it working with the generator when you try the Microtik do you see any different error messages? It is possible that Microtik data is a different format that is not compatible... That is why I wanted you to try the command above we should see some messages to help figure it out ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 24, 2023, 1:56pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/16 "2023-06-24T13:56:18Z")
> [@stephenb](#): > > filebeat -e -d "\*" ```auto {"log.level":"info","@timestamp":"2023-06-24T13:55:41.900Z","log.origin":{"file.name":"instance/beat.go","file.line":779},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:41.900Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":870},"message":"Beat metadata path: /var/lib/filebeat/meta.json","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-24T13:55:41.924Z","log.origin":{"file.name":"instance/beat.go","file.line":787},"message":"Beat ID: 4762b2c2-2127-44f1-9e4f-c0acd8bb1345","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:41.951Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition contains: map[]","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:41.951Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition !contains: map[]","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2023-06-24T13:55:44.952Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:44.987Z","log.logger":"docker","log.origin":{"file.name":"docker/client.go","file.line":49},"message":"Docker client will negotiate the API version on the first request.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:45.014Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":130},"message":"add_cloud_metadata: starting to fetch metadata, timeout=3s","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:46.952Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"add_docker_metadata/add_docker_metadata.go","file.line":93},"message":"add_docker_metadata: docker environment detected","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:46.952Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":213},"message":"Start docker containers scanner","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:46.952Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":375},"message":"List containers","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:47.005Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":266},"message":"Fetching events since 2023-06-24 13:55:47.005683912 +0000 UTC m=+5.312141392","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:47.071Z","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":148},"message":"Could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable","service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:48.063Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":174},"message":"add_cloud_metadata: timed-out waiting for all responses","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:48.063Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":133},"message":"add_cloud_metadata: fetchMetadata ran for 3.000541484s","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-24T13:55:48.063Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:48.063Z","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], condition=!contains: map[], add_cloud_metadata={}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.parent.pid]], add_kubernetes_metadata","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:48.063Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 4 times","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:48.480Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 3 times","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:48.881Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 2 times","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:49.281Z","log.origin":{"file.name":"locks/lock.go","file.line":79},"message":"Could not obtain lock for file /var/lib/filebeat/filebeat.lock, retrying 1 times","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-06-24T13:55:49.682Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":313},"message":"Watcher stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-06-24T13:55:49.682Z","log.origin":{"file.name":"instance/beat.go","file.line":426},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2023-06-24T13:55:49.682Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)","service.name":"filebeat","ecs.version":"1.6.0"} Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data) ``` ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [June 24, 2023, 2:01pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/17 "2023-06-24T14:01:54Z")
mikrotik mood ![microtick](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1a27a1ba373a08e1f8902e4c7840f78bd6b396b3.jpeg) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 24, 2023, 2:18pm UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/18 "2023-06-24T14:18:05Z")
> [@sana1567](#): > > ```auto > {"log.level":"error","@timestamp":"2023-06-24T13:55:49.682Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)","service.name":"filebeat","ecs.version":"1.6.0"} > Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data) > > ``` You have to stop the other filebeat before starting filebeat in the foreground with `filebeat -e -d "*"` You can only run 1 filebeat at a time... read the error messages they are pretty clear. Please try again... > [@sana1567](#): > > mikrotik mood I know nothing about mikrotik so I can not help there > `var.max_message_size` > The maximum size of the message received over UDP. The default is `10KiB`. I see 32K in your Mikrotik settings perhaps increase the max message size large in the netflow.yml to 32K or larger or decrease the size in the Mikrotik.... but that says cache so that may have nothing to do with it... Perhaps try a different format like version 9. Or perhaps look at some of the other setting in the filebeat netflow module > `var.read_buffer` > The size of the read buffer on the UDP socket. > `var.timeout` > The read and write timeout for socket operations. > `var.expiration_timeout` > The time before an idle session or unused template is expired. Only applicable to v9 and IPFIX protocols. A value of zero disables expiration. > `var.queue_size` > The maximum number of packets that can be queued for processing. Use this setting to avoid packet-loss when dealing with occasional bursts of traffic. Keep trying you are close. ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [July 1, 2023, 7:51am UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/19 "2023-07-01T07:51:07Z")
> [@stephenb](#): > > filebeat -e -d "\*" hello i stopped the filebeat service ```auto sudo systemctl stop filebeat.service sudo filebeat -e -d "*" ``` ```auto {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":137},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":223},"message":"Start next scan","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":324},"message":"Exclude file: /var/log/auth.log.2.gz","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":324},"message":"Exclude file: /var/log/syslog.2.gz","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":324},"message":"Exclude file: /var/log/auth.log.3.gz","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":324},"message":"Exclude file: /var/log/syslog.3.gz","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":324},"message":"Exclude file: /var/log/syslog.4.gz","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":324},"message":"Exclude file: /var/log/syslog.5.gz","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":324},"message":"Exclude file: /var/log/syslog.6.gz","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":324},"message":"Exclude file: /var/log/syslog.7.gz","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":471},"message":"Check file for harvesting: /var/log/syslog","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":569},"message":"Update existing file for harvesting: /var/log/syslog, offset: 618736","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog","state_id":"native::1048839-64768","finished":false,"os_id":"1048839-64768","old_source":"/var/log/syslog","old_finished":false,"old_os_id":"1048839-64768","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":622},"message":"Harvester for file is still running: /var/log/syslog","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog","state_id":"native::1048839-64768","finished":false,"os_id":"1048839-64768","old_source":"/var/log/syslog","old_finished":false,"old_os_id":"1048839-64768","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":324},"message":"Exclude file: /var/log/auth.log.4.gz","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":471},"message":"Check file for harvesting: /var/log/auth.log","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":471},"message":"Check file for harvesting: /var/log/syslog.1","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":569},"message":"Update existing file for harvesting: /var/log/auth.log, offset: 34104","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log","state_id":"native::1051372-64768","finished":false,"os_id":"1051372-64768","old_source":"/var/log/auth.log","old_finished":false,"old_os_id":"1051372-64768","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":622},"message":"Harvester for file is still running: /var/log/auth.log","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log","state_id":"native::1051372-64768","finished":false,"os_id":"1051372-64768","old_source":"/var/log/auth.log","old_finished":false,"old_os_id":"1051372-64768","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":471},"message":"Check file for harvesting: /var/log/auth.log.1","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":569},"message":"Update existing file for harvesting: /var/log/syslog.1, offset: 3875903","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog.1","state_id":"native::1048773-64768","finished":false,"os_id":"1048773-64768","old_source":"/var/log/syslog.1","old_finished":true,"old_os_id":"1048773-64768","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":624},"message":"File didn't change: /var/log/syslog.1","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog.1","state_id":"native::1048773-64768","finished":false,"os_id":"1048773-64768","old_source":"/var/log/syslog.1","old_finished":true,"old_os_id":"1048773-64768","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":287},"message":"input states cleaned up. Before: 2, After: 2, Pending: 0","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":569},"message":"Update existing file for harvesting: /var/log/auth.log.1, offset: 56884","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log.1","state_id":"native::1051117-64768","finished":false,"os_id":"1051117-64768","old_source":"/var/log/auth.log.1","old_finished":true,"old_os_id":"1051117-64768","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":624},"message":"File didn't change: /var/log/auth.log.1","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log.1","state_id":"native::1051117-64768","finished":false,"os_id":"1051117-64768","old_source":"/var/log/auth.log.1","old_finished":true,"old_os_id":"1051117-64768","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:50.560Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":287},"message":"input states cleaned up. Before: 2, After: 2, Pending: 0","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:55.560Z","log.logger":"input.harvester","log.origin":{"file.name":"log/log.go","file.line":111},"message":"End of file reached: /var/log/auth.log; Backoff now.","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log","state_id":"native::1051372-64768","finished":false,"os_id":"1051372-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051372-64768","harvester_id":"362e2d85-fca4-4f40-ab33-787120f6701f","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:55.575Z","log.logger":"input.harvester","log.origin":{"file.name":"log/log.go","file.line":111},"message":"End of file reached: /var/log/syslog; Backoff now.","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog","state_id":"native::1048839-64768","finished":false,"os_id":"1048839-64768","old_source":"/var/log/syslog","old_finished":true,"old_os_id":"1048839-64768","harvester_id":"8f4c6d42-fc7e-4ee8-94ef-9036f5cc548c","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:55.575Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"add_docker_metadata/add_docker_metadata.go","file.line":150},"message":"Error while extracting container ID from source path: index is out of range for field 'log.file.path'","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:55.575Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n \"@timestamp\": \"2023-07-01T07:45:55.575Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.8.0\",\n \"pipeline\": \"filebeat-8.8.0-system-syslog-pipeline\"\n },\n \"host\": {\n \"ip\": [\n \"192.168.10.63\",\n \"fe80::20c:29ff:fea6:6eb9\",\n \"172.17.0.1\",\n \"fe80::ecee:eeff:feee:eeee\",\n \"10.1.131.128\",\n \"fe80::64e0:54ff:fe3b:a22c\"\n],\n \"mac\": [\n \"00-0C-29-A6-6E-B9\",\n \"02-42-CC-CE-D2-B8\",\n \"66-E0-54-3B-A2-2C\",\n \"EE-EE-EE-EE-EE-EE\"\n],\n \"name\": \"syslog\",\n \"hostname\": \"syslog\",\n \"architecture\": \"x86_64\",\n \"os\": {\n \"type\": \"linux\",\n \"platform\": \"ubuntu\",\n \"version\": \"20.04.6 LTS (Focal Fossa)\",\n \"family\": \"debian\",\n \"name\": \"Ubuntu\",\n \"kernel\": \"5.4.0-152-generic\",\n \"codename\": \"focal\"\n },\n \"id\": \"053d81efd282406a9b6c3c51d7f2e141\",\n \"containerized\": false\n },\n \"service\": {\n \"type\": \"system\"\n },\n \"input\": {\n \"type\": \"log\"\n },\n \"event\": {\n \"dataset\": \"system.syslog\",\n \"module\": \"system\",\n \"timezone\": \"+00:00\"\n },\n \"ecs\": {\n \"version\": \"1.12.0\"\n },\n \"agent\": {\n \"name\": \"syslog\",\n \"type\": \"filebeat\",\n \"version\": \"8.8.0\",\n \"ephemeral_id\": \"5723ea32-8b9a-498b-9841-d2370777c2de\",\n \"id\": \"4762b2c2-2127-44f1-9e4f-c0acd8bb1345\"\n },\n \"log\": {\n \"file\": {\n \"path\": \"/var/log/syslog\"\n },\n \"offset\": 618736\n },\n \"message\": \"Jul 1 07:45:49 syslog systemd[1]: run-containerd-runc-k8s.io-1c3413b54a206c2fbe54d6ce215fd021939f77ca1efd6b35c01be0ba91d4f366-runc.N0z7zw.mount: Succeeded.\",\n \"fileset\": {\n \"name\": \"syslog\"\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:56.575Z","log.logger":"input.harvester","log.origin":{"file.name":"log/log.go","file.line":111},"message":"End of file reached: /var/log/syslog; Backoff now.","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog","state_id":"native::1048839-64768","finished":false,"os_id":"1048839-64768","old_source":"/var/log/syslog","old_finished":true,"old_os_id":"1048839-64768","harvester_id":"8f4c6d42-fc7e-4ee8-94ef-9036f5cc548c","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:56.603Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":264},"message":"PublishEvents: 1 events have been published to elasticsearch in 27.229182ms.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:56.603Z","log.logger":"acker","log.origin":{"file.name":"beater/acker.go","file.line":59},"message":"stateful ack","service.name":"filebeat","count":1,"ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:56.603Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":81},"message":"ackloop: return ack to broker loop:1","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:56.603Z","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":83},"message":"ackloop: done send ack","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:56.603Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":262},"message":"Processing 1 events","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:56.603Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":229},"message":"Registrar state updates processed. Count: 1","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.603Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":205},"message":"Registry file updated. 4 active states.","service.name":"filebeat","ecs.version":"1.6.0"} ^C{"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"service","log.origin":{"file.name":"service/service.go","file.line":52},"message":"Received signal \"interrupt\", stopping","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.origin":{"file.name":"beater/filebeat.go","file.line":460},"message":"Stopping filebeat","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.origin":{"file.name":"beater/crawler.go","file.line":155},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.origin":{"file.name":"beater/crawler.go","file.line":165},"message":"Stopping 0 inputs","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":227},"message":"Dynamic config reloader stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":139},"message":"Stopping 2 runners ...","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":150},"message":"Stopping runner: system (auth), system (syslog)","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":150},"message":"Stopping runner: netflow (log)","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.origin":{"file.name":"input/input.go","file.line":134},"message":"input ticker stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.origin":{"file.name":"input/input.go","file.line":134},"message":"input ticker stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"netflow","log.origin":{"file.name":"netflow/input.go","file.line":188},"message":"Stopping UDP input","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"UDP","log.origin":{"file.name":"dgram/server.go","file.line":131},"message":"Stopping datagram socket server for UDP","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"udp","log.origin":{"file.name":"dgram/handler.go","file.line":68},"message":"Connection has been closed","service.name":"filebeat","address":"0.0.0.0:2055","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":145},"message":"client: closing acker","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":150},"message":"client: done closing acker","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":152},"message":"client: close queue producer","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":176},"message":"client: cancelled 0 events","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":155},"message":"client: done producer close","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":158},"message":"client: closing processors","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":163},"message":"client: done closing processors","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":152},"message":"Stopped runner: netflow (log)","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"reader_multiline","log.origin":{"file.name":"multiline/pattern.go","file.line":142},"message":"Multiline event flushed because timeout reached.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":340},"message":"End of file reached. Closing because close_eof is enabled.","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log","state_id":"native::1051372-64768","finished":false,"os_id":"1051372-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051372-64768","harvester_id":"362e2d85-fca4-4f40-ab33-787120f6701f","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":616},"message":"Stopping harvester.","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log","state_id":"native::1051372-64768","finished":false,"os_id":"1051372-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051372-64768","harvester_id":"362e2d85-fca4-4f40-ab33-787120f6701f","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":626},"message":"Closing file","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log","state_id":"native::1051372-64768","finished":false,"os_id":"1051372-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051372-64768","harvester_id":"362e2d85-fca4-4f40-ab33-787120f6701f","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":490},"message":"Update state (offset: 34104).","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log","state_id":"native::1051372-64768","finished":false,"os_id":"1051372-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051372-64768","harvester_id":"362e2d85-fca4-4f40-ab33-787120f6701f","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":637},"message":"harvester cleanup finished.","service.name":"filebeat","input_id":"7ff66fd0-29e1-42d8-94e4-363d77bd69aa","source_file":"/var/log/auth.log","state_id":"native::1051372-64768","finished":false,"os_id":"1051372-64768","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"1051372-64768","harvester_id":"362e2d85-fca4-4f40-ab33-787120f6701f","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":145},"message":"client: closing acker","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":150},"message":"client: done closing acker","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":152},"message":"client: close queue producer","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":176},"message":"client: cancelled 0 events","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":155},"message":"client: done producer close","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":158},"message":"client: closing processors","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.766Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":163},"message":"client: done closing processors","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.766Z","log.origin":{"file.name":"input/input.go","file.line":134},"message":"input ticker stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.767Z","log.origin":{"file.name":"harvester/forwarder.go","file.line":52},"message":"Input outlet closed","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":616},"message":"Stopping harvester.","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog","state_id":"native::1048839-64768","finished":false,"os_id":"1048839-64768","old_source":"/var/log/syslog","old_finished":true,"old_os_id":"1048839-64768","harvester_id":"8f4c6d42-fc7e-4ee8-94ef-9036f5cc548c","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":626},"message":"Closing file","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog","state_id":"native::1048839-64768","finished":false,"os_id":"1048839-64768","old_source":"/var/log/syslog","old_finished":true,"old_os_id":"1048839-64768","harvester_id":"8f4c6d42-fc7e-4ee8-94ef-9036f5cc548c","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":490},"message":"Update state (offset: 618893).","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog","state_id":"native::1048839-64768","finished":false,"os_id":"1048839-64768","old_source":"/var/log/syslog","old_finished":true,"old_os_id":"1048839-64768","harvester_id":"8f4c6d42-fc7e-4ee8-94ef-9036f5cc548c","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":637},"message":"harvester cleanup finished.","service.name":"filebeat","input_id":"e25aa39d-e144-4346-bd5a-cff125d63f60","source_file":"/var/log/syslog","state_id":"native::1048839-64768","finished":false,"os_id":"1048839-64768","old_source":"/var/log/syslog","old_finished":true,"old_os_id":"1048839-64768","harvester_id":"8f4c6d42-fc7e-4ee8-94ef-9036f5cc548c","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":145},"message":"client: closing acker","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":150},"message":"client: done closing acker","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":152},"message":"client: close queue producer","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":176},"message":"client: cancelled 0 events","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":155},"message":"client: done producer close","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":158},"message":"client: closing processors","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":163},"message":"client: done closing processors","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":152},"message":"Stopped runner: system (auth), system (syslog)","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.767Z","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.767Z","log.origin":{"file.name":"beater/filebeat.go","file.line":460},"message":"Stopping filebeat","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":132},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":166},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":167},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.767Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":137},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.770Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":195},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"user.slice","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"user.slice","total":{"ns":1814448528}},"memory":{"id":"session-1.scope","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":275218432}}}},"cpu":{"system":{"ticks":60,"time":{"ms":60}},"total":{"ticks":170,"time":{"ms":170},"value":170},"user":{"ticks":110,"time":{"ms":110}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":13},"info":{"ephemeral_id":"5723ea32-8b9a-498b-9841-d2370777c2de","name":"filebeat","uptime":{"ms":45750},"version":"8.8.0"},"memstats":{"gc_next":20918808,"memory_alloc":11631400,"memory_sys":33404168,"memory_total":58758632,"rss":93659136},"runtime":{"goroutines":25}},"filebeat":{"events":{"active":0,"added":67,"done":67},"harvester":{"closed":2,"open_files":0,"running":0,"skipped":0,"started":2},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":2,"starts":2,"stops":0},"reloads":1,"scans":1},"output":{"batches":{"split":0},"events":{"acked":61,"active":0,"batches":8,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":61},"read":{"bytes":30221,"errors":0},"type":"elasticsearch","write":{"bytes":85385,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":6,"published":61,"retry":48,"total":67},"queue":{"acked":61,"max_events":4096}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"registrar":{"states":{"cleanup":0,"current":4,"update":67},"writes":{"fail":0,"success":8,"total":8}},"system":{"cpu":{"cores":4},"load":{"1":0.31,"15":1.58,"5":0.85,"norm":{"1":0.0775,"15":0.395,"5":0.2125}}}},"ecs.version":"1.6.0"}} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.770Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":196},"message":"Uptime: 45.857201301s","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.770Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":163},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2023-07-01T07:45:57.770Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":313},"message":"Watcher stopped","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2023-07-01T07:45:57.770Z","log.origin":{"file.name":"instance/beat.go","file.line":528},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"} ``` ---
**Author:** ![sana1567](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1567/32/122093_2.png) [@sana1567](https://discuss.elastic.co/u/sana1567)\ **Post date:** [July 1, 2023, 9:51am UTC](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692/20 "2023-07-01T09:51:10Z")
finally the statistics worked, opened YouTube ```auto www.youtube.com/watch?v=D4KBu4Oc3Jw&ab_channel=EvermightTech ``` did the configuration like his and it worked thank you very much for your help!!! [Next page](https://discuss.elastic.co/t/netflow-mikrotik-no-data-in-elasticsearch/335692.md?page=2)