Running Windows 10, Logstash 8.1.0, Elasticsearch, kibana and filebeat 8.0.0 all on the same machine.
Getting the data from logstash into elasticsearch and kibana works but when multiple files get read at the same time the files with multiple lines are seen as only one record.
here is the pipeline which is in C:\Program Files\logstash-8.1.0\first-pipeline.conf:
input {
beats {
port => "5044"
}
}
filter {
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{DATESTAMP:DateTime},%{PName:Program},%{POSINT:ProcessID},%{POSINT:UsageCPU}" }
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
user => "elastic"
password => "my password"
ssl => true
ssl_certificate_verification => false
# The name of the Index
index => "<logstash_filelog>"
}
}
If new entries get into the file while the pipeline and filebeat are active they will be set as new hits in the discover tab but when they all get read for example the next day they are only marked as one record even though there are multiple lines.
Hey, thanks for the quick answer here is my filebeat.yml:
filebeat.inputs:
# filestream is an input for collecting log messages from files.
- type: filestream
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- C:\TEMP\filebeatlog\*.log
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list. example: ['^ERR', '^WARN']
include_lines: [A-Za-z0-9]
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
username: "elastic"
password: "my password"
the rest stayed default did not change anything. I saw some multiline commands in the reference.yml maybe I should include them like this?
# The maximum number of lines that are combined to one event.
# In case there are more the max_lines the additional lines are discarded.
# Default is 500
multiline.max_lines: 1
# The number of lines to aggregate into a single event.
multiline.count_lines: 1
It seems like default is 500 maybe thats the reason why all these lines get seen as one?
No, the multline.* settings only has effect if you are using multine, what you aren't from the filebeat.yml you shared.
Since you do not have multline configured and filebeat sent all the lines of the file as a single event, then it is probably the source file that is this way.
Hey thanks for you answer,
a simple delphi application which logs every 20 seconds random numbers and names out of a range in a line nothing more.
You can see the structure of the log file at the top.
log.flags is set to multiline. Is this something I am able to control or change? Shouldnt the input.type be filestream and not "log" as seen in the screenshot?
Yes, if you were running a filestream collector then the type would be filestream, not log. Perhaps someone else is running a second filebeat instance?
Can you share the log file that you're trying to read with Filebeat? (File Upload, not a copy&paste here).
You didn't set any multiline configuration (good) so the only reason that can cause Filebeat to send a document with multiple lines in "message" is that Filebeat is not recognizing the line terminator in your file.
Alternatively, just try adding the line_terminator option to the filestream config
Hey, thanks for your answer.
Im not allowed to upload the log here ("Sorry, the file you are trying to upload is not authorized (authorized extensions: jpg, jpeg, png, gif).")
I have uploaded it to Google Drive here:
Unfortunately when using line_terminator: carriage_return
I get the same problem with it being flagged as multiline
This part of your config is wrong. You need to surround the regular expression with quotes, like this:
include_lines: '[A-Za-z0-9]'
I guess you have this fixed in your setup, otherwise you wouldn't be ingesting any data due to this error. If that's not the case, then definitely this must be another Filebeat instance running with multiline settings.
Please post your current configuration, first remove the line_terminator: test as it is not necessary with your data.
@kvch, can you help here? Can this be caused by file rotation?
Well, that's weird, because when I tested with that error, I was getting no data ingested (all events were filtered) instead of the full file being ingested as a single document.
Yeah it is quite weird. If I remember correctly it was without the ' ' in the filebeat reference documentation so I didnt think this was the issue but apparently it was.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.