Elasticsearch-does-not-send-logs-to-filebeat-and-or-logstash

Elastic Stack Beats
1

I have input but cant show output! I need help!

filebeat.yml

filebeat.inputs:

  • type: filestream
    id: router1
    enabled: true
    paths:
    • /u01/redis_app/run/tomcat-deploy-camidlog/logs/report_log/report.log
      output.logstash:
      hosts: ["10.79.7.121:5045"]

test output:

[root@openid121 report_log]# /usr/share/filebeat/bin/filebeat test output -c /etc/filebeat/filebeatlog.yml
logstash: 10.79.7.121:5045...
connection...
parse host... OK
dns lookup... OK
addresses: 10.79.7.121
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK

log filebeat

2022-07-19T18:06:44.034+0700 INFO instance/beat.go:685 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/var/log/filebeatlog] Hostfs Path: [/]
2022-07-19T18:06:44.034+0700 INFO instance/beat.go:693 Beat ID: 15decd44-10a9-44a0-9e70-b60baa0016df
2022-07-19T18:06:44.035+0700 INFO [seccomp] seccomp/seccomp.go:101 Syscall filter could not be installed because the kernel does not support seccomp
2022-07-19T18:06:44.035+0700 INFO [beat] instance/beat.go:1039 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/var/log/filebeatlog"}, "type": "filebeat", "uuid": "15decd44-10a9-44a0-9e70-b60baa0016df"}}}
2022-07-19T18:06:44.035+0700 INFO [beat] instance/beat.go:1048 Build info {"system_info": {"build": {"commit": "1993ee88a11cb34f61a1fb45c7c3cf50533682cb", "libbeat": "7.17.3", "time": "2022-04-19T09:27:20.000Z", "version": "7.17.3"}}}
2022-07-19T18:06:44.035+0700 INFO [beat] instance/beat.go:1051 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.8"}}}
2022-07-19T18:06:44.036+0700 INFO [beat] instance/beat.go:1055 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-11-06T00:06:01+07:00","containerized":false,"name":"openid121","ip":["127.0.0.1/8","::1/128","10.79.94.121/24","fe80::2a6e:d4ff:fe88:c7ea/64"],"kernel_version":"3.10.0-514.el7.x86_64","mac":["28:6e:d4:88:c7:ea"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":3,"patch":1611,"codename":"Core"},"timezone":"ICT","timezone_offset_sec":25200,"id":"2161603f9ea14bab9ad24b14d8f10d71"}}}
2022-07-19T18:06:44.036+0700 INFO [beat] instance/beat.go:1084 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/u01/redis_app/run/tomcat-deploy-camidapi/logs/report_log", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 40985, "ppid": 1, "seccomp": {"mode":"disabled"}, "start_time": "2022-07-19T18:06:43.290+0700"}}}
2022-07-19T18:06:44.036+0700 INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.17.3
2022-07-19T18:06:44.042+0700 INFO [publisher] pipeline/module.go:113 Beat name: openid121
2022-07-19T18:06:44.043+0700 WARN beater/filebeat.go:202 Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-07-19T18:06:44.043+0700 INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
2022-07-19T18:06:44.043+0700 INFO instance/beat.go:492 filebeat start running.
2022-07-19T18:06:44.043+0700 INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2022-07-19T18:06:44.043+0700 INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2022-07-19T18:06:44.043+0700 WARN beater/filebeat.go:411 Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-07-19T18:06:44.043+0700 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2022-07-19T18:06:44.043+0700 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2022-07-19T18:06:44.043+0700 INFO [crawler] beater/crawler.go:117 starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]
2022-07-19T18:06:44.043+0700 INFO [crawler] beater/crawler.go:148 Starting input (ID: 6004394033242872904)
2022-07-19T18:06:44.044+0700 INFO [crawler] beater/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2022-07-19T18:06:44.044+0700 INFO [input.filestream] compat/compat.go:111 Input filestream starting {"id": "router1"}
2022-07-19T18:06:44.044+0700 INFO [file_watcher] filestream/fswatch.go:138 Start next scan
2022-07-19T18:06:54.045+0700 INFO [file_watcher] filestream/fswatch.go:138 Start next scan
2022-07-19T18:07:04.044+0700 INFO [file_watcher] filestream/fswatch.go:138 Start next scan
2022-07-19T18:07:14.044+0700 INFO [file_watcher] filestream/fswatch.go:138 Start next scan
2022-07-19T18:07:14.051+0700 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":59}},"total":{"ticks":190,"time":{"ms":206},"value":190},"user":{"ticks":140,"time":{"ms":147}}},"handles":{"limit":{"hard":1000000,"soft":1000000},"open":12},"info":{"ephemeral_id":"90cf46b3-2414-4478-9a91-66f6468a662c","uptime":{"ms":30068},"version":"7.17.3"},"memstats":{"gc_next":18474688,"memory_alloc":10726064,"memory_sys":34161672,"memory_total":55564688,"rss":108060672},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":0.55,"15":0.5,"5":0.52,"norm":{"1":0.0688,"15":0.0625,"5":0.065}}}}}}

2

Are you already check whether the ip and port you're using now in filebeat and logstash are correct?

3

Hi, could you please format your post first using code tags?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.