Hi guys I'm new to the Elastic stack and I'm currently trying to setup a quick and easy pipeline between Logstash and Filebeats where filebeat will send a test log file and logstash will display it on stdout. Currently nothing is being displayed, and I have no idea why, whether if the error is occuring on filebeat's configuration, or logstash's configuration. I have all the information attached below with my log files and their output info.
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- home/ubuntu/Xombo/Logstash/filebeat/logs/logstash-tutorial.log
output.logstash:
hosts: ["localhost:5044"]
filebeats output info:
2021-02-02T14:46:43.874-0800 INFO instance/beat.go:645 Home path: [/home/ubuntu/Xombo/Logstash/filebeat] Config path: [/home/ubuntu/Xombo/Logstash/filebeat] Data path: [/home/ubuntu/Xombo/Logstash/filebeat/data] Logs path: [/home/ubuntu/Xombo/Logstash/filebeat/logs]
2021-02-02T14:46:43.874-0800 INFO instance/beat.go:653 Beat ID: f82ec680-2407-4de6-96c6-7624e0b6f96a
2021-02-02T14:46:46.876-0800 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2021-02-02T14:46:46.876-0800 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-02-02T14:46:46.876-0800 INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/home/ubuntu/Xombo/Logstash/filebeat", "data": "/home/ubuntu/Xombo/Logstash/filebeat/data", "home": "/home/ubuntu/Xombo/Logstash/filebeat", "logs": "/home/ubuntu/Xombo/Logstash/filebeat/logs"}, "type": "filebeat", "uuid": "f82ec680-2407-4de6-96c6-7624e0b6f96a"}}}
2021-02-02T14:46:46.876-0800 INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T23:11:24.000Z", "version": "7.10.2"}}}
2021-02-02T14:46:46.876-0800 INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.12"}}}
2021-02-02T14:46:46.877-0800 INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-02T09:16:56-08:00","containerized":false,"name":"ubuntu","ip":["127.0.0.1/8","::1/128","10.0.2.15/24","fe80::ee76:36a9:5724:6ee3/64"],"kernel_version":"5.4.0-58-generic","mac":["08:00:27:72:d3:86"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.1 LTS (Focal Fossa)","major":20,"minor":4,"patch":1,"codename":"focal"},"timezone":"PST","timezone_offset_sec":-28800,"id":"8639ab5acdeb4be6a874e751802c4dd7"}}}
2021-02-02T14:46:46.877-0800 INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/ubuntu/Xombo/Logstash/filebeat", "exe": "/home/ubuntu/Xombo/Logstash/filebeat/filebeat", "name": "filebeat", "pid": 12778, "ppid": 12777, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-02-02T14:46:43.680-0800"}}}
2021-02-02T14:46:46.878-0800 INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.10.2
2021-02-02T14:46:46.878-0800 INFO [publisher] pipeline/module.go:113 Beat name: ubuntu
2021-02-02T14:46:46.879-0800 WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-02-02T14:46:46.880-0800 INFO instance/beat.go:455 filebeat start running.
2021-02-02T14:46:46.880-0800 INFO memlog/store.go:119 Loading data file of '/home/ubuntu/Xombo/Logstash/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-02-02T14:46:46.880-0800 INFO memlog/store.go:124 Finished loading transaction log file for '/home/ubuntu/Xombo/Logstash/filebeat/data/registry/filebeat'. Active transaction id=0
2021-02-02T14:46:46.880-0800 WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-02-02T14:46:46.880-0800 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2021-02-02T14:46:46.880-0800 INFO [crawler] beater/crawler.go:71 Loading Inputs: 2
2021-02-02T14:46:46.881-0800 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0
2021-02-02T14:46:46.881-0800 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-02-02T14:46:46.881-0800 INFO cfgfile/reload.go:164 Config reloader started
2021-02-02T14:46:46.881-0800 INFO cfgfile/reload.go:224 Loading of config files completed.
logstash config file
input {
beats {
port => "5044"
host => "localhost"
}
stdin {}
}
output {
stdout { codec => rubydebug }
}
logstash output info
Using bundled JDK: /home/ubuntu/Xombo/Logstash/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-12721/jruby5260584726433080772jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /home/ubuntu/Xombo/Logstash/logstash/logs which is now configured via log4j2.properties
[2021-02-02T14:45:58,868][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[2021-02-02T14:45:59,610][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-02-02T14:46:02,037][INFO ][org.reflections.Reflections] Reflections took 31 ms to scan 1 urls, producing 23 keys and 47 values
[2021-02-02T14:46:02,700][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/home/ubuntu/Xombo/Logstash/logstash/logstash-sample.conf"], :thread=>"#<Thread:0x57398de2 run>"}
[2021-02-02T14:46:03,600][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.86}
[2021-02-02T14:46:03,627][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2021-02-02T14:46:03,734][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2021-02-02T14:46:03,829][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-02-02T14:46:03,899][INFO ][org.logstash.beats.Server][main][da93a1980417833ad39d18e06c5708bfa28c631cebd68688ea74aa5ba70fb07b] Starting server on port: 5044
[2021-02-02T14:46:04,131][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Any help will be appreciated, and also sorry if the formatting is weird or wrong since this is my first time posting on here as well.