Filebeats not sending logs to Logstash to be viewed in stdout

Hi guys I'm new to the Elastic stack and I'm currently trying to setup a quick and easy pipeline between Logstash and Filebeats where filebeat will send a test log file and logstash will display it on stdout. Currently nothing is being displayed, and I have no idea why, whether if the error is occuring on filebeat's configuration, or logstash's configuration. I have all the information attached below with my log files and their output info.

filebeat.yml

filebeat.inputs:
- type: log

  enabled: true

  paths:
    - home/ubuntu/Xombo/Logstash/filebeat/logs/logstash-tutorial.log

output.logstash:
  hosts: ["localhost:5044"]

filebeats output info:

2021-02-02T14:46:43.874-0800	INFO	instance/beat.go:645	Home path: [/home/ubuntu/Xombo/Logstash/filebeat] Config path: [/home/ubuntu/Xombo/Logstash/filebeat] Data path: [/home/ubuntu/Xombo/Logstash/filebeat/data] Logs path: [/home/ubuntu/Xombo/Logstash/filebeat/logs]
2021-02-02T14:46:43.874-0800	INFO	instance/beat.go:653	Beat ID: f82ec680-2407-4de6-96c6-7624e0b6f96a
2021-02-02T14:46:46.876-0800	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2021-02-02T14:46:46.876-0800	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2021-02-02T14:46:46.876-0800	INFO	[beat]	instance/beat.go:981	Beat info	{"system_info": {"beat": {"path": {"config": "/home/ubuntu/Xombo/Logstash/filebeat", "data": "/home/ubuntu/Xombo/Logstash/filebeat/data", "home": "/home/ubuntu/Xombo/Logstash/filebeat", "logs": "/home/ubuntu/Xombo/Logstash/filebeat/logs"}, "type": "filebeat", "uuid": "f82ec680-2407-4de6-96c6-7624e0b6f96a"}}}
2021-02-02T14:46:46.876-0800	INFO	[beat]	instance/beat.go:990	Build info	{"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T23:11:24.000Z", "version": "7.10.2"}}}
2021-02-02T14:46:46.876-0800	INFO	[beat]	instance/beat.go:993	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.12"}}}
2021-02-02T14:46:46.877-0800	INFO	[beat]	instance/beat.go:997	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-02T09:16:56-08:00","containerized":false,"name":"ubuntu","ip":["127.0.0.1/8","::1/128","10.0.2.15/24","fe80::ee76:36a9:5724:6ee3/64"],"kernel_version":"5.4.0-58-generic","mac":["08:00:27:72:d3:86"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.1 LTS (Focal Fossa)","major":20,"minor":4,"patch":1,"codename":"focal"},"timezone":"PST","timezone_offset_sec":-28800,"id":"8639ab5acdeb4be6a874e751802c4dd7"}}}
2021-02-02T14:46:46.877-0800	INFO	[beat]	instance/beat.go:1026	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/ubuntu/Xombo/Logstash/filebeat", "exe": "/home/ubuntu/Xombo/Logstash/filebeat/filebeat", "name": "filebeat", "pid": 12778, "ppid": 12777, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-02-02T14:46:43.680-0800"}}}
2021-02-02T14:46:46.878-0800	INFO	instance/beat.go:299	Setup Beat: filebeat; Version: 7.10.2
2021-02-02T14:46:46.878-0800	INFO	[publisher]	pipeline/module.go:113	Beat name: ubuntu
2021-02-02T14:46:46.879-0800	WARN	beater/filebeat.go:178	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-02-02T14:46:46.880-0800	INFO	instance/beat.go:455	filebeat start running.
2021-02-02T14:46:46.880-0800	INFO	memlog/store.go:119	Loading data file of '/home/ubuntu/Xombo/Logstash/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-02-02T14:46:46.880-0800	INFO	memlog/store.go:124	Finished loading transaction log file for '/home/ubuntu/Xombo/Logstash/filebeat/data/registry/filebeat'. Active transaction id=0
2021-02-02T14:46:46.880-0800	WARN	beater/filebeat.go:381	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-02-02T14:46:46.880-0800	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2021-02-02T14:46:46.880-0800	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 2
2021-02-02T14:46:46.881-0800	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2021-02-02T14:46:46.881-0800	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2021-02-02T14:46:46.881-0800	INFO	cfgfile/reload.go:164	Config reloader started
2021-02-02T14:46:46.881-0800	INFO	cfgfile/reload.go:224	Loading of config files completed.

logstash config file

input {
    beats {
        port => "5044"
        host => "localhost"
    }

    stdin {}
}

output {
    stdout { codec => rubydebug }
}

logstash output info

Using bundled JDK: /home/ubuntu/Xombo/Logstash/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-12721/jruby5260584726433080772jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /home/ubuntu/Xombo/Logstash/logstash/logs which is now configured via log4j2.properties
[2021-02-02T14:45:58,868][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[2021-02-02T14:45:59,610][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-02-02T14:46:02,037][INFO ][org.reflections.Reflections] Reflections took 31 ms to scan 1 urls, producing 23 keys and 47 values 
[2021-02-02T14:46:02,700][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/home/ubuntu/Xombo/Logstash/logstash/logstash-sample.conf"], :thread=>"#<Thread:0x57398de2 run>"}
[2021-02-02T14:46:03,600][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.86}
[2021-02-02T14:46:03,627][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2021-02-02T14:46:03,734][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2021-02-02T14:46:03,829][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-02-02T14:46:03,899][INFO ][org.logstash.beats.Server][main][da93a1980417833ad39d18e06c5708bfa28c631cebd68688ea74aa5ba70fb07b] Starting server on port: 5044
[2021-02-02T14:46:04,131][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Any help will be appreciated, and also sorry if the formatting is weird or wrong since this is my first time posting on here as well.

Hi @Juunis, welcome to the Elastic community forums!

For problems like this one involving Filebeat and Logstash, I would suggest trying to narrow the scope of the problem first. Let's try to figure out if the problem is happening on the Filebeat side or the Logstash side.

To do this, temporarily disable the Logstash output in your Filebeat configuration. You can do this by adding enabled: false under the output.logstash: section.

Then, enable the Console output in your Filebeat configuration.

Then, run Filebeat again and check if you see your events on STDOUT from the Filebeat process.

  • If you do, then the problem is not in Filebeat's ingestion of your log file; it must be either on the Logstash side or somewhere between Filebeat and Logstash.
  • If you don't, then the problem is on Filebeat's side.

Once we've narrowed the scope of the problem, we can proceed further.

Hope that helps,

Shaunak

Hey @shaunak ,

Thanks for the reply! I've tried what you told me by adding a output.console section and adding enabling:true to it as well but I didn't get any output on the console. Does this mean the filebeats is not reading the logs from the path I specified?

Thanks!

That is what it would seem like. Please run filebeat export config and note the value of path.data — it will be a folder. It contains, amongst other things, a registry file where Filebeat maintains state of which log files it is reading or has read and some statistics about them. Delete that folder and run Filebeat again while keeping the Console output enabled. See if that makes any difference. Please also post the entire Filebeat log from this run here so I can see what happened.

Thanks,

Shaunak

I've deleted the registry folder and here's the output after I've deleted it and maintained the console output:

2021-02-03T09:27:24.159-0800	INFO	instance/beat.go:645	Home path: [/home/ubuntu/Xombo/Logstash/filebeat] Config path: [/home/ubuntu/Xombo/Logstash/filebeat] Data path: [/home/ubuntu/Xombo/Logstash/filebeat/data] Logs path: [/home/ubuntu/Xombo/Logstash/filebeat/logs]
2021-02-03T09:27:24.160-0800	INFO	instance/beat.go:653	Beat ID: f82ec680-2407-4de6-96c6-7624e0b6f96a
2021-02-03T09:27:27.164-0800	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2021-02-03T09:27:27.164-0800	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2021-02-03T09:27:27.164-0800	INFO	[beat]	instance/beat.go:981	Beat info	{"system_info": {"beat": {"path": {"config": "/home/ubuntu/Xombo/Logstash/filebeat", "data": "/home/ubuntu/Xombo/Logstash/filebeat/data", "home": "/home/ubuntu/Xombo/Logstash/filebeat", "logs": "/home/ubuntu/Xombo/Logstash/filebeat/logs"}, "type": "filebeat", "uuid": "f82ec680-2407-4de6-96c6-7624e0b6f96a"}}}
2021-02-03T09:27:27.164-0800	INFO	[beat]	instance/beat.go:990	Build info	{"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T23:11:24.000Z", "version": "7.10.2"}}}
2021-02-03T09:27:27.164-0800	INFO	[beat]	instance/beat.go:993	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.12"}}}
2021-02-03T09:27:27.165-0800	INFO	[beat]	instance/beat.go:997	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-03T09:14:17-08:00","containerized":false,"name":"ubuntu","ip":["127.0.0.1/8","::1/128","10.0.2.15/24","fe80::ee76:36a9:5724:6ee3/64"],"kernel_version":"5.4.0-58-generic","mac":["08:00:27:72:d3:86"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.1 LTS (Focal Fossa)","major":20,"minor":4,"patch":1,"codename":"focal"},"timezone":"PST","timezone_offset_sec":-28800,"id":"8639ab5acdeb4be6a874e751802c4dd7"}}}
2021-02-03T09:27:27.165-0800	INFO	[beat]	instance/beat.go:1026	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/ubuntu/Xombo/Logstash/filebeat", "exe": "/home/ubuntu/Xombo/Logstash/filebeat/filebeat", "name": "filebeat", "pid": 2273, "ppid": 2272, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-02-03T09:27:21.660-0800"}}}
2021-02-03T09:27:27.165-0800	INFO	instance/beat.go:299	Setup Beat: filebeat; Version: 7.10.2
2021-02-03T09:27:27.166-0800	INFO	[publisher]	pipeline/module.go:113	Beat name: ubuntu
2021-02-03T09:27:27.166-0800	WARN	beater/filebeat.go:178	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-02-03T09:27:27.167-0800	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2021-02-03T09:27:27.167-0800	INFO	instance/beat.go:455	filebeat start running.
2021-02-03T09:27:27.183-0800	INFO	memlog/store.go:119	Loading data file of '/home/ubuntu/Xombo/Logstash/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-02-03T09:27:27.183-0800	INFO	memlog/store.go:124	Finished loading transaction log file for '/home/ubuntu/Xombo/Logstash/filebeat/data/registry/filebeat'. Active transaction id=0
2021-02-03T09:27:27.183-0800	WARN	beater/filebeat.go:381	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-02-03T09:27:27.183-0800	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2021-02-03T09:27:27.183-0800	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 2
2021-02-03T09:27:27.184-0800	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2021-02-03T09:27:27.184-0800	INFO	cfgfile/reload.go:164	Config reloader started
2021-02-03T09:27:27.184-0800	INFO	cfgfile/reload.go:224	Loading of config files completed.
2021-02-03T09:27:57.168-0800	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"id":"user@1000.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":1316003840}}}},"cpu":{"system":{"ticks":60,"time":{"ms":60}},"total":{"ticks":150,"time":{"ms":152},"value":150},"user":{"ticks":90,"time":{"ms":92}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":10},"info":{"ephemeral_id":"62ea051e-c967-4d19-8a2b-a74c7fa39764","uptime":{"ms":34166}},"memstats":{"gc_next":19139664,"memory_alloc":10555944,"memory_total":44206928,"rss":72175616},"runtime":{"goroutines":19}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0.85,"15":0.87,"5":0.93,"norm":{"1":0.425,"15":0.435,"5":0.465}}}}}}

Oh I just look at the output from the config command and it seems my inputs are not enabled even though I have enabled as true. Do you have any clue as to why this might be?

filebeat.yml:

filebeat.inputs:

- type: log

  enabled: true

  paths:
    - home/ubuntu/Xombo/Logstash/filebeat/logs/logstash-tutorial.log

- type: filestream

  enabled: true

  paths:
    - /var/log/*.log

export config output:

filebeat:
  config:
    modules:
      path: /home/ubuntu/Xombo/Logstash/filebeat/modules.d/*.yml
      reload:
        enabled: false
  inputs:
  - enabled: false
    paths:
    - /home/ubuntu/Xombo/Logstash/logs/logstash-tutorial.log
    type: log
  - enabled: false
    paths:
    - /var/log/*.log
    type: filestream
output:
  logstash:
    hosts:
    - localhost:5044
path:
  config: /home/ubuntu/Xombo/Logstash/filebeat
  data: /home/ubuntu/Xombo/Logstash/filebeat/data
  home: /home/ubuntu/Xombo/Logstash/filebeat
  logs: /home/ubuntu/Xombo/Logstash/filebeat/logs
processors:
- add_host_metadata:
    when:
      not:
        contains:
          tags: forwarded
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup:
  kibana: null
  template:
    settings:
      index:
        number_of_shards: 1

That is a good find (about the inputs being disabled). I can't see anything in your source config that would cause this. I copy-pasted your config into my own file and the filebeat export config command shows the inputs as enabled for me.

This makes me think there are some YAML formatting issues in play here.

Can you try the same thing that I did: copy-paste the source config block from your last post into a new file, add the output.console.enabled: true setting to it at the end and then try running filebeat export config with that source config?

Shaunak

Hey I found out the issue. It seems the yaml file I was looking at and editing was a different yaml file from a different folder but I thought it was the folder that was being used when starting filebeat. Thats why the inputs were disabled. Once I figured that out and enabled the inputs, everything worked! thanks for helping me! It really saved me a lot of time since I probably wouldn't have figured all this out.

Thanks!!!

Patrick Chu