Filebeat invalid template id

Elastic Stack Beats
1

Hi,
I have some issue in filebeat. I had sent nProbe netflow v9 to filebeat but when I start filebeat it comes the problem below.

elklab@localhost filebeat]$ sudo filebeat -e
[sudo] password for elklab:
2020-10-19T17:51:24.067+0800 INFO instance/beat.go:621 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-10-19T17:51:24.067+0800 INFO instance/beat.go:629 Beat ID: d5cd54a7-e74d-43b9-af3a-efd1c89a98a9
2020-10-19T17:51:24.069+0800 INFO [seccomp] seccomp/seccomp.go:101 Syscall filter could not be installed because the kernel does not support seccomp
2020-10-19T17:51:24.069+0800 INFO [beat] instance/beat.go:957 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "d5cd54a7-e74d-43b9-af3a-efd1c89a98a9"}}}
2020-10-19T17:51:24.069+0800 INFO [beat] instance/beat.go:966 Build info {"system_info": {"build": {"commit": "5e69e25b920e3d93bec76a09a31da3ab35a55607", "libbeat": "7.7.0", "time": "2020-05-12T00:53:16.000Z", "version": "7.7.0"}}}
2020-10-19T17:51:24.070+0800 INFO [beat] instance/beat.go:969 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.9"}}}
2020-10-19T17:51:24.070+0800 INFO [beat] instance/beat.go:973 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-10-13T15:08:25+08:00","containerized":false,"name":"localhost.localdomain","ip":["127.0.0.1/8","::1/128","192.168.0.45/24","fe80::b626:edb1:9854:b35f/64","192.168.122.1/24"],"kernel_version":"3.10.0-693.el7.x86_64","mac":["00:0c:29:18:5d:78","52:54:00:c8:a1:f4","52:54:00:c8:a1:f4"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":4,"patch":1708,"codename":"Core"},"timezone":"CST","timezone_offset_sec":28800,"id":"55770b2df47d4662a25873f7ba5bc2db"}}}
2020-10-19T17:51:24.070+0800 INFO [beat] instance/beat.go:1002 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 29739, "ppid": 29733, "seccomp": {"mode":"disabled"}, "start_time": "2020-10-19T17:51:23.819+0800"}}}
2020-10-19T17:51:24.070+0800 INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.7.0
2020-10-19T17:51:24.071+0800 INFO [publisher] pipeline/module.go:110 Beat name: localhost.localdomain
2020-10-19T17:51:24.071+0800 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-10-19T17:51:24.071+0800 INFO instance/beat.go:438 filebeat start running.
2020-10-19T17:51:24.071+0800 WARN beater/filebeat.go:335 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-10-19T17:51:24.071+0800 INFO registrar/registrar.go:145 Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json
2020-10-19T17:51:24.071+0800 INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-10-19T17:51:24.071+0800 INFO beater/crawler.go:73 Loading Inputs: 1
2020-10-19T17:51:24.071+0800 INFO beater/crawler.go:105 Loading and starting Inputs completed. Enabled inputs: 0
2020-10-19T17:51:24.071+0800 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-10-19T17:51:24.071+0800 INFO cfgfile/reload.go:175 Config reloader started
2020-10-19T17:51:27.069+0800 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-10-19T17:51:34.072+0800 INFO input/input.go:114 Starting input of type: netflow; ID: 6782364051398218258
2020-10-19T17:51:34.072+0800 INFO [netflow] netflow/input.go:153 Starting UDP input
2020-10-19T17:51:34.073+0800 INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "192.168.0.45:2055"}
2020-10-19T17:51:37.583+0800 WARN [netflow] netflow/input.go:244 Error parsing NetFlow packet of length 192 from 192.168.1.96:40038: error parsing set: invalid template id
2020-10-19T17:51:38.583+0800 INFO [publisher_pipeline_output] pipeline/output.go:101 Connecting to backoff(async(tcp://0.0.0.0:5044))
2020-10-19T17:51:38.584+0800 INFO [publisher_pipeline_output] pipeline/output.go:111 Connection to backoff(async(tcp://0.0.0.0:5044)) established
2020-10-19T17:51:42.012+0800 WARN [netflow] netflow/input.go:244 Error parsing NetFlow packet of length 192 from 192.168.1.96:38407: error parsing set: invalid template id
2020-10-19T17:51:46.140+0800 WARN [netflow] netflow/input.go:244 Error parsing NetFlow packet of length 192 from 192.168.1.96:40038: error parsing set: invalid template id
2020-10-19T17:51:52.378+0800 WARN [netflow] netflow/input.go:244 Error parsing NetFlow packet of length 192 from 192.168.1.96:38407: error parsing set: invalid template id

And here is my filebeat.yml configuration.
#=========================== Filebeat inputs =============================

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

  • type: netflow
    host: "0.0.0.0:2055"
    protocols: [v9]

    #Change to true to enable this input configuration.
    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /var/log/*.log
      #- c:\programdata\elasticsearch\logs*

Is there anyone have idea?

2

Could you please format your configuration with </>?

3

When you start up Filebeat it has to receive the Netflow template records from the source device before it can know how to parse the Netflow flow records. So there will be a period where it cannot parse flows, but after the device sends the templates it should be good.

Most sources allow you to configure how often the templates are sent (like 30s, 1m, 5m, etc).

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.