Filebeat Netflow module doesn't load

Elastic Stack Beats
,
1

hi

ive installed filebeat ver 7.7.0 using arm repository.
udp input and logstash output work fine.
only when im configuring netflow input filebeat fail to start.
appreciate your help.

bellow is the out put of debug:
indent preformatted text by 4 spaces

th: [/usr/share/filebeat/bin/logs]
2020-06-16T11:36:01.188+0300 INFO instance/beat.go:629 Beat ID: 708a290c-86e3-45ae-9209-c3a5f8745ff6
2020-06-16T11:36:01.189+0300 INFO [beat] instance/beat.go:957 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat/bin", "data": "/usr/share/filebeat/bin/data", "home": "/usr/share/filebeat/bin", "logs": "/usr/share/filebeat/bin/logs"}, "type": "filebeat", "uuid": "708a290c-86e3-45ae-9209-c3a5f8745ff6"}}}
2020-06-16T11:36:01.189+0300 INFO [beat] instance/beat.go:966 Build info {"system_info": {"build": {"commit": "unknown", "libbeat": "7.7.0", "time": "1754-08-30T22:43:41.128Z", "version": "7.7.0"}}}
2020-06-16T11:36:01.189+0300 INFO [beat] instance/beat.go:969 Go runtime info {"system_info": {"go": {"os":"linux","arch":"arm64","max_procs":4,"version":"go1.14.3"}}}
2020-06-16T11:36:01.191+0300 INFO [beat] instance/beat.go:973 Host info {"system_info": {"host": {"architecture":"aarch64","boot_time":"2020-06-01T08:27:13+03:00","containerized":false,"name":"filebeat-M54","ip":["127.0.0.1/8","::1/128","172.17.200.41/24","fe80::dea6:32ff:fe17:ff99/64"],"kernel_version":"5.3.0-1026-raspi2","mac":["dc:a6:32:17:ff:99","dc:a6:32:17:ff:9b"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.4 LTS (Bionic Beaver)","major":18,"minor":4,"patch":4,"codename":"bionic"},"timezone":"IDT","timezone_offset_sec":10800,"id":"8d827af341ba49cbab6669e90ed6e0d9"}}}
2020-06-16T11:36:01.192+0300 INFO [beat] instance/beat.go:1002 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/ubuntu", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 13558, "ppid": 13537, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-06-16T11:36:00.520+0300"}}}
2020-06-16T11:36:01.192+0300 INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.7.0
2020-06-16T11:36:01.194+0300 INFO [publisher] pipeline/module.go:110 Beat name: filebeat-M54
2020-06-16T11:36:01.197+0300 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-06-16T11:36:01.197+0300 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-06-16T11:36:01.198+0300 INFO instance/beat.go:438 filebeat start running.
2020-06-16T11:36:01.198+0300 WARN beater/filebeat.go:335 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-06-16T11:36:01.198+0300 INFO registrar/registrar.go:145 Loading registrar data from /var/lib/filebeat/filebeat/data.json
2020-06-16T11:36:01.198+0300 INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-06-16T11:36:01.198+0300 INFO beater/crawler.go:73 Loading Inputs: 2
2020-06-16T11:36:01.199+0300 INFO input/input.go:114 Starting input of type: udp; ID: 13441375058660919524
2020-06-16T11:36:01.199+0300 INFO udp/input.go:103 Starting UDP input
2020-06-16T11:36:01.199+0300 INFO beater/crawler.go:138 Stopping Crawler
2020-06-16T11:36:01.199+0300 INFO beater/crawler.go:148 Stopping 1 inputs
2020-06-16T11:36:01.199+0300 INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "0.0.0.0:9000"}
2020-06-16T11:36:01.199+0300 INFO input/input.go:149 input ticker stopped
2020-06-16T11:36:01.199+0300 INFO input/input.go:167 Stopping Input: 13441375058660919524
2020-06-16T11:36:01.199+0300 INFO udp/input.go:118 Stopping UDP input
2020-06-16T11:36:01.199+0300 INFO [udp] udp/server.go:140 Stopping UDP server {"address": "0.0.0.0:9000"}
2020-06-16T11:36:01.201+0300 INFO [udp] udp/server.go:118 Connection has been closed {"address": "0.0.0.0:9000"}
2020-06-16T11:36:01.201+0300 INFO [udp] udp/server.go:144 UDP server stopped {"address": "0.0.0.0:9000"}
2020-06-16T11:36:01.201+0300 INFO beater/crawler.go:164 Crawler stopped
2020-06-16T11:36:01.201+0300 INFO registrar/registrar.go:367 Stopping Registrar
2020-06-16T11:36:01.201+0300 INFO registrar/registrar.go:293 Ending Registrar
2020-06-16T11:36:01.221+0300 INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":63}},"total":{"ticks":150,"time":{"ms":159},"value":150},"user":{"ticks":90,"time":{"ms":96}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":8},"info":{"ephemeral_id":"695764ea-2857-4a1b-b327-caeb643b7d9b","uptime":{"ms":92}},"memstats":{"gc_next":27099440,"memory_alloc":13557840,"memory_total":18348864,"rss":33075200},"runtime":{"goroutines":12}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}
2020-06-16T11:36:01.222+0300 INFO [monitoring] log/log.go:154 Uptime: 100.77255ms
2020-06-16T11:36:01.223+0300 INFO [monitoring] log/log.go:131 Stopping metrics logging.
2020-06-16T11:36:01.223+0300 INFO instance/beat.go:444 filebeat stopped.
2020-06-16T11:36:01.223+0300 ERROR instance/beat.go:932 Exiting: Error while initializing input: Error creating input. No such input type exist: 'netflow'
Exiting: Error while initializing input: Error creating input. No such input type exist: 'netflow'

2

Hi,

What filebeat package have you used, and from what arm repository?

Netflow input is in x-pack, and not included in OSS builds.

3

hi
thanks for replaying
i don't really know if it xpack or oss.
i do know that Netflow folder was included in module.d dir so i assume it the Xpack.
this is the repo ive used

echo "deb https://raw.githubusercontent.com/RaoulDuke-Esq/Beats-Pi/master buster main" | sudo tee -a /etc/apt/sources.list.d/beats-pi.list
4

For the size of the packages they look like x-pack ones, yes, but I cannot say with certainty.
In any case "netflow" input should work i this is an x-pack build. Could you share the configuration where you are trying to use it?

5

hi
attached my config.
this configuration works on other systems that are not arm based

thanks
indent preformatted text by 4 spaces
filebeat.inputs:

  • type: udp
    max_message_size: 10MiB
    host: "0.0.0.0:9000"
    fields:
    logzio_codec: plain

    Your Logz.io account token. You can find your token at

    https://app.logz.io/#/dashboard/settings/manage-accounts

    token: xxxxxxxxxxxxxxxxxxxxxxxxx
    type: network-device
    fields_under_root: true
    encoding: utf-8
    ignore_older: 3h
  • type: netflow
    max_message_size: 10KiB
    host: "0.0.0.0:2054"
    protocols: [ v5, v9, ipfix ]
    fields:
    logzio_codec: plain

    Your Logz.io account token. You can find your token at

    https://app.logz.io/#/dashboard/settings/manage-accounts

    token: xxxxxxxxxxxxxxxxxxxxxxxxxx
    type: netflow
    fields_under_root: true
    expiration_timeout: 30m
    queue_size: 8192
    tags: ["netflow"]

    ... For Filebeat 7 only ...

filebeat.registry.path: /var/lib/filebeat
processors:
#- rename:
# fields:
output.logstash:
hosts: ["listener.logz.io:5015"]
ssl:
certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

indent preformatted text by 4 spaces
6

Could you try with one of the packages built by Elastic CI? The ones for 7.7 are available here: https://beats-ci.elastic.co/job/Beats/job/packaging/job/7.7/14/gcsObjects/

7

ooo great

i assume you refer to this one filebeat-7.7.2-SNAPSHOT-arm64.deb
right?
ill try that

8

it didnt work, ive got this error
indent preformatted text by 4 spaces
dpkg-deb: error: 'filebeat-7.7.2-SNAPSHOT-arm64.deb' is not a Debian format archive
indent preformatted text by 4 spaces
should i use a different package?

9

Did you use the [Download] link to download this package?

This is the link for the debian package of filebeat for arm64, just in case: https://storage.cloud.google.com/beats-ci-artifacts/snapshots/filebeat-7.7.2-SNAPSHOT-arm64.deb

10

hi
sorry for the delay.
yes, I've used the download and I verify that the package was downloaded.
I've tried again the link you've added, but still the same issue.

indent preformatted text by 4 spaces

filebeat-7.7.2-SNAPSHOT-arm64.deb
root@ubuntu:~# dpkg -i filebeat-7.7.2-SNAPSHOT-arm64.deb
dpkg-deb: error: 'filebeat-7.7.2-SNAPSHOT-arm64.deb' is not a Debian format archive
dpkg: error processing archive filebeat-7.7.2-SNAPSHOT-arm64.deb (--install):
dpkg-deb --control subprocess returned error exit status 2
Errors were encountered while processing:
filebeat-7.7.2-SNAPSHOT-arm64.deb
indent preformatted text by 4 spaces

11

How are you downloading these files? These URLs need to be authenticated with a google account. If you are donwloading them with some CLI tool you may be downloading the login page.

Try with this URL, that doesn't require authentication: https://storage.googleapis.com/beats-ci-artifacts/snapshots/filebeat-7.7.2-SNAPSHOT-amd64.deb

12

great, now it works!!

is this package is a formal package of elastic?
are they maintain by you guys?

thanks

13

These packages are built by elastic as part of the testing process, but they are not officially supported.

By the way, I have just seen that I sent you the link for the amd64 build, did you find the package for arm64?

14

yes i did
thanks

1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.