Filebeat not shipping files

Elastic Stack Beats
1

I am migrating from a very old version of the logstash shipper to Filebeat but it won't enable my log prospector.

018-07-12T15:30:32.002+0100 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 0

I just can see where the problem lies. The file exists, it is under 24 hours old and enabled is set to true. I am only outputting to Redis. Logstash and Elasticsearch have been commented out.

The important parts of my config are as follows:

  • type: log

    Change to true to enable this input configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /a/logfile/location/that/exists

    Exclude lines. A list of regular expressions to match. It drops the lines that are

    matching any regular expression from the list.

    #exclude_lines: ['^DBG']

    Include lines. A list of regular expressions to match. It exports the lines that are

    matching any regular expression from the list.

    #include_lines: ['^ERR', '^WARN']

    Exclude files. A list of regular expressions to match. Filebeat drops the files that

    are matching any regular expression from the list. By default, no files are dropped.

    #exclude_files: ['.gz$']

    Optional additional fields. These fields can be freely picked

    to add additional information to the crawled log files for filtering

    #fields:

    level: debug

    review: 1

    Multiline options

    Mutiline can be used for log messages spanning multiple lines. This is common

    for Java Stack Traces or C-Line Continuation

    The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

    multiline.pattern: ^%{TIMESTAMP_ISO8601}

    Defines if the pattern set under pattern should be negated or not. Default is false.

    multiline.negate: true

    Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

    that was (not) matched before or after or as long as a pattern is not matched based on negate.

    Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

    multiline.match: after

and I am outputting to Redis

output:
redis:
hosts: ["host:6380"]
save_topology: true
redis.key: "list1"
db: 0
db_topology: 1
timeout: 5
reconnect_interval: 1

2

I think I have solved it. It would be useful if it actually had a prospectors entry. When I added to following line in bold it worked. I can't really understand why it isn't there in the first place.

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

filebeat.prospectors:

  • type: log

    Change to true to enable this input configuration.

    enabled: false

    Paths that should be crawled and fetched. Glob based paths.

    paths:

3

Unfortunately the formatting does not really make it visible what you changed. Please reformat and use 3 ticks before and after your config parts.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.