Can filebeat load non .log file into elasticsearch/logstash?

Elastic Stack Beats
1

Hi there.
I try to load some files from filebeat -> logstash -> elasticsearch -> kibana.
The thing is that the files I'm trying to load are not .log files, but files without any extension.
the filebeat.yml is:

filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • input_type: log

    Paths that should be crawled and fetched. Glob based paths.

    paths:

- /var/log/*.log

- /path/to/files/*/subdir/*

when I touch in the subdir folder a file with .log extension kibana immediately recognize it.
What do I need to do in order to see in kibana the files that don't have .log extention? should I use filebeat at all?

2

Well, extension or not doesn't matter. What matters is:

  • How is the file written too (append only?), or is file ever changed/updated after being written
  • Is the file in binary or plain text?
3

Hi

  1. The file is written and being appended. What I'm trying to do is to read the file as it being updated.
  2. The file is a text file.
4

Filebeat splits by newline characters \n or \r\n.

If the producer really uses append (some tools don't append, but create a copy + new contents appended) AND newline is used to separate lines/entries, sure you can use filebeat to load the files.

Just make sure the glob pattern can find the files. Have you checked the files are actually found by filebeat, by having a look at filebeat logs (consider enabling debug logging)?

Please properly format logs and configs using the </> buttong. At first it wasn't really clear to me you actually posted part of your config. Beats configs are sensitive to indentation. Without proper formatting it's hard to see potential errors.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.