Hello World for filebeat

Elastic Stack Beats
1

I am attempting to do a 'hello-world' with filebeat by reading from a file and outputing to stdout or another file. I find that filebeat does not detect any changes to the log files.

What would a hello-world look like?

What I've tried:

Attempt 1:

/etc/filebeat/filebeat.yml

---
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
filebeat.input:
- type: log
  enabled: true
  paths:
    - /tmp/input.log
output.console:
  pretty: true

I'm starting the service like so:

filebeat -e

Then putting content into the log file

echo "foobar" >> /tmp/intput.log

I find no output in the console

Attempt 2

I've also attempted to send the output to a new file, with no luck

/etc/filebeat/filebeat.yml

---
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
filebeat.input:
- type: log
  enabled: true
  paths:
    - /tmp/input.log
output.file:
  path: "/tmp/"
  filename: foobar

The output from the filebeat server

instance/beat.go:292	Setup Beat: filebeat; Version: 7.3.1
2019-09-03T21:24:45.120Z	INFO	[publisher]	pipeline/module.go:97	Beat name: ubuntu-bionic
2019-09-03T21:24:45.121Z	WARN	beater/filebeat.go:152	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-09-03T21:24:45.122Z	INFO	instance/beat.go:421	filebeat start running.
2019-09-03T21:24:45.124Z	INFO	registrar/registrar.go:145	Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json
2019-09-03T21:24:45.125Z	INFO	registrar/registrar.go:152	States Loaded from registrar: 0
2019-09-03T21:24:45.126Z	WARN	beater/filebeat.go:368	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-09-03T21:24:45.127Z	INFO	crawler/crawler.go:72	Loading Inputs: 1
2019-09-03T21:24:45.128Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 0
2019-09-03T21:24:45.129Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2019-09-03T21:24:45.130Z	INFO	cfgfile/reload.go:171	Config reloader started
2019-09-03T21:24:45.130Z	INFO	cfgfile/reload.go:226	Loading of config files completed.
2019-09-03T21:24:48.120Z	INFO	add_cloud_metadata/add_cloud_metadata.go:347	add_cloud_metadata: hosting provider type not detected.

How can I configure filebeat to send data from one location to another?

2

Hello, thanks for reaching out regarding filebeat. Which operating system and filebeat version are you running?

I would suggest trying to explicitly specify the filebeat config file with the -c flag. It may also be useful to enable debugging output.

The following provides examples of both settings:
https://www.elastic.co/guide/en/beats/filebeat/current/enable-filebeat-debugging.html

3

Typo?

echo "foobar" >> /tmp/input.log

4

I've repeated the above examples 3 times. Once on OSX, once on the official filebeat docker image, and once with an ubuntu 18.04 vagrant vm. All 3 have the same result where filebeat fails to read from a file.

I've tried using globs like this:

filebeat.input:
- type: log
  enabled: true
  paths:
    - /tmp/*.log

I've also tried referencing the file directly

filebeat.input:
- type: log
  enabled: true
  paths:
    - /tmp/input.log

And yes, I've verified I'm not typing the file name

cat /tmp/input.log
foo
bar
foobar
heyo

I've also tried using logs in a different directory since some OS's use a symlink for /tmp

filebeat.input:
- type: log
  enabled: true
  paths:
    - /var/log/input.log

Filebeat fails to read the file in all situations.

I am starting filebeat like so:

filebeat -e

I've also tried ensuring filebeat is using the correct config file

filebeat -e --path.config /etc/filebeat

I've also tried not using -e and running filebeat as normal

filebeat --path.config /etc/filebeat
5

EDIT is that still the output section show the output section of filebeat.yml

Also what version are you running and it appears you have modules enabled from the first log, what modules did you enable? I am not sure modules plus output to console is supported since the use ingest pipelines which looks like perhaps the error from the the original post.

7

for reading symlinks please use the following setting
# If symlinks is enabled, symlinks are opened and harvested. The harvester is opening the
# original for harvesting but will report the symlink name as source.
symlinks: true

make sure that only console output is enabled and other outputs (Kafka, Redis, Logstash, ElasticSearch, etc ) are disabled

8

While I never did get the file.output working, I was able to get output.console working.

There was a typo in the configs:

file.input:

Should be

file.inputs

The full working config is:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/foobar.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  enabled: false
output.console:
  enabled: true
  pretty: true
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
9

Cool

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.