Hello,
I have a strange issue. I'm extracting json file from a server and my filebeat is meant to send them to logstash and then Kibana.
Here is the thing: The files sent aren't being taken into account by logstash when added to the directory, I have to modify them (adding a " " or a random character in the file) so that logstash start to process them and send them to kibana. Even stranger, i did a script that add a space at the beginning of all the files in the directory wanted, which, if I do it myself, make them processed by logstash, but here it doesn't. After I check, the spaces are indeed at the beginning of the files.
Here is my filebeat.yml
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /home/test/Documents/PRODUCERS/Data/HC/*.json
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
document_type: json
### Multiline options
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#multiline.pattern: '{"id":'
# Defines if the pattern set under pattern should be negated or not. Default is false.
multiline.negate: true
#ignore_older: 5m
#max_backoff: 1s
#scan_frequency: 15s
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
multiline.match: before
#fields_under_root: true
#close_inactive: 8m
#clean_inactive: 10m
#clean_removed: true