Filebeat can't send any log to logstash. No error can be found from the debug mode. Any one knows what is the issue?
Here is the filebeat.yml configuration:
filebeat.inputs:
- type: log
tags: ["test1"]
document_type: local
enabled: true
paths:- /var/log/files/log/test1.local
output.logstash:
hosts: ["logstash:5443"]
ssl.certificate_authorities: ["/etc/ssl/certs/ca.crt"]
The debug output is:
root@pc51:/etc/filebeat# filebeat -e -v -c ./filebeat.yml -d ""
2020-05-06T13:44:57.413-0700 INFO instance/beat.go:622 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-05-06T13:44:57.413-0700 DEBUG [beat] instance/beat.go:674 Beat metadata path: /var/lib/filebeat/meta.json
2020-05-06T13:44:57.413-0700 INFO instance/beat.go:630 Beat ID: 56a89347-0a56-4bcb-a3ba-2d139389b59e
2020-05-06T13:44:57.415-0700 DEBUG [docker] docker/client.go:48 Docker client will negotiate the API version on the first request.
2020-05-06T13:44:57.415-0700 DEBUG [filters] add_cloud_metadata/providers.go:126 add_cloud_metadata: starting to fetch metadata, timeout=3s
2020-05-06T13:44:57.425-0700 DEBUG [add_docker_metadata] add_docker_metadata/add_docker_metadata.go:91 add_docker_metadata: docker environment detected
2020-05-06T13:44:57.426-0700 DEBUG [docker] docker/watcher.go:198 Start docker containers scanner
2020-05-06T13:44:57.426-0700 DEBUG [docker] docker/watcher.go:338 List containers
2020-05-06T13:44:57.428-0700 DEBUG [bus] bus/bus.go:83 docker: map[container:0xc000616150 start:true]
2020-05-06T13:44:57.428-0700 DEBUG [docker] docker/watcher.go:246 Fetching events since 1588797897
2020-05-06T13:44:57.429-0700 DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:109 add_kubernetes_metadata: could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided
2020-05-06T13:45:00.416-0700 DEBUG [filters] add_cloud_metadata/providers.go:169 add_cloud_metadata: timed-out waiting for all responses
2020-05-06T13:45:00.416-0700 DEBUG [filters] add_cloud_metadata/providers.go:129 add_cloud_metadata: fetchMetadata ran for 3.000618063s
2020-05-06T13:45:00.416-0700 INFO add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-05-06T13:45:00.416-0700 DEBUG [processors] processors/processor.go:101 Generated new processors: add_host_metadata=[netinfo.enabled=[false], cache.ttl=[5m0s]], add_cloud_metadata=null, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.ppid]], add_kubernetes_metadata
2020-05-06T13:45:00.416-0700 DEBUG [seccomp] seccomp/seccomp.go:117 Loading syscall filter {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-05-06T13:45:00.417-0700 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-05-06T13:45:00.417-0700 INFO [beat] instance/beat.go:958 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "56a89347-0a56-4bcb-a3ba-2d139389b59e"}}}
2020-05-06T13:45:00.417-0700 INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "d57bcf8684602e15000d65b75afcd110e2b12b59", "libbeat": "7.6.2", "time": "2020-03-26T05:23:38.000Z", "version": "7.6.2"}}}
2020-05-06T13:45:00.417-0700 INFO [beat] instance/beat.go:970 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.8"}}}
2020-05-06T13:45:00.418-0700 INFO [beat] instance/beat.go:974 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-04-09T22:08:10-07:00","containerized":false,"name":"pc51","ip":["127.0.0.1/8","::1/128","10.32.0.51/16","fe80::250:56ff:feb1:b2fd/64","172.17.0.1/16","fe80::42:c6ff:fe2c:1ca8/64","fe80::3cbd:acff:fe4a:4128/64"],"kernel_version":"4.15.0-91-generic","mac":["00:50:56:b1:94:15","00:50:56:b1:b2:fd","02:42:c6:2c:1c:a8","3e:bd:ac:4a:41:28"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.4 LTS (Bionic Beaver)","major":18,"minor":4,"patch":4,"codename":"bionic"},"timezone":"PDT","timezone_offset_sec":-25200,"id":"bde3b1b76d1e4c1d9a86f8e174820b94"}}}
2020-05-06T13:45:00.418-0700 INFO [beat] instance/beat.go:1003 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 6708, "ppid": 11033, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-05-06T13:44:57.130-0700"}}}
2020-05-06T13:45:00.418-0700 INFO instance/beat.go:298 Setup Beat: filebeat; Version: 7.6.2
2020-05-06T13:45:00.418-0700 DEBUG [beat] instance/beat.go:324 Initializing output plugins
2020-05-06T13:45:00.419-0700 DEBUG [tls] tlscommon/tls.go:155 successfully loaded CA certificate: /etc/ssl/certs/ca.crt
2020-05-06T13:45:00.419-0700 DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer
2020-05-06T13:45:00.419-0700 INFO [publisher] pipeline/module.go:110 Beat name: pc51
2020-05-06T13:45:00.420-0700 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-05-06T13:45:00.420-0700 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-05-06T13:45:00.420-0700 INFO instance/beat.go:439 filebeat start running.
2020-05-06T13:45:00.420-0700 DEBUG [test] registrar/migrate.go:159 isFile(/var/lib/filebeat/registry) -> false
2020-05-06T13:45:00.420-0700 DEBUG [test] registrar/migrate.go:159 isFile() -> false
2020-05-06T13:45:00.420-0700 DEBUG [test] registrar/migrate.go:152 isDir(/var/lib/filebeat/registry/filebeat) -> true
2020-05-06T13:45:00.420-0700 DEBUG [test] registrar/migrate.go:159 isFile(/var/lib/filebeat/registry/filebeat/meta.json) -> true
2020-05-06T13:45:00.420-0700 DEBUG [registrar] registrar/migrate.go:51 Registry type '0' found
2020-05-06T13:45:00.420-0700 DEBUG [registrar] registrar/registrar.go:125 Registry file set to: /var/lib/filebeat/registry/filebeat/data.json
2020-05-06T13:45:00.420-0700 INFO registrar/registrar.go:145 Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json
2020-05-06T13:45:00.420-0700 INFO registrar/registrar.go:152 States Loaded from registrar: 1
2020-05-06T13:45:00.420-0700 WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-05-06T13:45:00.420-0700 INFO crawler/crawler.go:72 Loading Inputs: 1
2020-05-06T13:45:00.421-0700 DEBUG [input] log/config.go:204 recursive glob enabled
2020-05-06T13:45:00.421-0700 DEBUG [input] log/input.go:164 exclude_files: []. Number of stats: 1
2020-05-06T13:45:00.421-0700 DEBUG [input] log/input.go:185 input with previous states loaded: 0
2020-05-06T13:45:00.421-0700 INFO log/input.go:152 Configured paths: [/var/log/files/log/test1.local]
2020-05-06T13:45:00.421-0700 INFO input/input.go:114 Starting input of type: log; ID: 5428102665616574311
2020-05-06T13:45:00.421-0700 DEBUG [cfgfile] cfgfile/reload.go:137 Checking module configs from: /etc/filebeat/modules.d/.yml
2020-05-06T13:45:00.421-0700 DEBUG [cfgfile] cfgfile/reload.go:151 Number of module configs found: 0
2020-05-06T13:45:00.421-0700 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-05-06T13:45:00.421-0700 DEBUG [registrar] registrar/registrar.go:278 Starting Registrar
2020-05-06T13:45:00.421-0700 DEBUG [input] log/input.go:191 Start next scan
2020-05-06T13:45:00.421-0700 DEBUG [input] log/input.go:212 input states cleaned up. Before: 0, After: 0, Pending: 0
2020-05-06T13:45:00.421-0700 INFO cfgfile/reload.go:175 Config reloader started
2020-05-06T13:45:00.421-0700 DEBUG [cfgfile] cfgfile/reload.go:205 Scan for new config files
2020-05-06T13:45:00.421-0700 DEBUG [cfgfile] cfgfile/reload.go:224 Number of module configs found: 0
2020-05-06T13:45:00.421-0700 DEBUG [reload] cfgfile/list.go:62 Starting reload procedure, current runners: 0
2020-05-06T13:45:00.422-0700 DEBUG [reload] cfgfile/list.go:80 Start list: 0, Stop list: 0
2020-05-06T13:45:00.422-0700 INFO cfgfile/reload.go:235 Loading of config files completed.
2020-05-06T13:45:10.421-0700 DEBUG [input] input/input.go:152 Run input
2020-05-06T13:45:10.421-0700 DEBUG [input] log/input.go:191 Start next scan
2020-05-06T13:45:10.422-0700 DEBUG [input] log/input.go:212 input states cleaned up. Before: 0, After: 0, Pending: 0