Hey there,
I am configuring several Beats with a custom mapping. I would just to know if there is another way to load custom template (setup.template.*) at startup or I must use the fields.yml file. I mean, if I have hundreds of fields I have to specify all of them into my custom fields.yml file. Not really easy I guess.
Apart from specifying a fields using a fields.yml file you have two more options. You can also use setup.template.append_fields to add custom fields to the default template. Alternatively, if you want to load a completely custom template you can use setup.template.json.* settings to configure a JSON template.
See more: Configure Elasticsearch index template loading | Heartbeat Reference [8.1] | Elastic
I tried to use setup.template.json.* but nothing happened.
No index template is created and taking a look to Filebeat logs (enabling them to DEBUG level) I can just see a warning message related to legacy/composable index template.
DEPRECATED: Please migrate your JSON templates from legacy template format to composable index template. Will be removed in version: 8.0.0
If I try to manually create index template, using the exact content of my json file, it works properly.
This is my filebeat.yml configuration:
logging.level: debug
setup.template.json.enabled: true
setup.template.json.path: "template.json"
setup.template.json.name: "customtemplate"
filebeat.inputs:
- type: filestream
paths:
- /usr/share/filebeat/input_data/*.csv
output.elasticsearch:
...
...
...Could you please share the debug logs?
This is the Filebeat logs:
2022-04-15T10:02:11.593Z INFO instance/beat.go:686 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs] Hostfs Path: [/]
2022-04-15T10:02:11.593Z DEBUG [beat] instance/beat.go:744 Beat metadata path: /usr/share/filebeat/data/meta.json
2022-04-15T10:02:11.597Z INFO instance/beat.go:694 Beat ID: 82b90266-41e7-4df4-a383-275ffc9ca118
2022-04-15T10:02:11.602Z WARN [cfgwarn] template/config.go:88 DEPRECATED: Please migrate your JSON templates from legacy template format to composable index template. Will be removed in version: 8.0.0
2022-04-15T10:02:11.604Z DEBUG [seccomp] seccomp/seccomp.go:117 Loading syscall filter {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","clone3","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2022-04-15T10:02:11.604Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2022-04-15T10:02:11.605Z INFO [beat] instance/beat.go:1040 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "82b90266-41e7-4df4-a383-275ffc9ca118"}}}
2022-04-15T10:02:11.605Z INFO [beat] instance/beat.go:1049 Build info {"system_info": {"build": {"commit": "1d05ba86138cfc9a5ae5c0acc64a57b8d81678ff", "libbeat": "7.17.1", "time": "2022-02-23T23:38:04.000Z", "version": "7.17.1"}}}
2022-04-15T10:02:11.605Z INFO [beat] instance/beat.go:1052 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.17.6"}}}
2022-04-15T10:02:11.607Z INFO [beat] instance/beat.go:1056 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-04-12T21:18:03Z","containerized":false,"name":"filebeat","ip":["127.0.0.1/8","172.18.0.2/16"],"kernel_version":"5.10.104-linuxkit","mac":["02:42:ac:12:00:02"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.3 LTS (Focal Fossa)","major":20,"minor":4,"patch":3,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0}}}
2022-04-15T10:02:11.607Z INFO [beat] instance/beat.go:1085 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 7, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-04-15T10:02:10.620Z"}}}
2022-04-15T10:02:11.607Z INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.17.1
2022-04-15T10:02:11.608Z DEBUG [beat] instance/beat.go:354 Initializing output plugins
2022-04-15T10:02:11.608Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'seclog' as ILM is enabled.
2022-04-15T10:02:11.608Z WARN [cfgwarn] tlscommon/config.go:100 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2022-04-15T10:02:11.611Z DEBUG [tls] tlscommon/tls.go:172 Successfully loaded CA certificate: /usr/share/filebeat/ca.crt
2022-04-15T10:02:11.611Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://elasticsearch:9200
2022-04-15T10:02:11.612Z DEBUG [tls] tlscommon/tls.go:172 Successfully loaded CA certificate: /usr/share/filebeat/ca.crt
2022-04-15T10:02:11.613Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-04-15T10:02:11.614Z DEBUG [publisher] pipeline/consumer.go:148 start pipeline event consumer
2022-04-15T10:02:11.614Z INFO [publisher] pipeline/module.go:113 Beat name: filebeat
2022-04-15T10:02:11.622Z INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
2022-04-15T10:02:11.623Z INFO instance/beat.go:492 filebeat start running.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.