Problem understanding the concept of filebeat -> logstash

[sd2j]

Well I got everything running .. at least partially. I am able to get logs from filebeat -> logstash -> elastic -> kibana

I have to admit i haven't read all documentation - which is way to much for my short time. But I seem to have an understanding problem.

I am using logstash as an "connection point". It receives data on a port and pushes it to elasticsearch - I am aware that filebeat is able to push directly to elastic but if there is a untypical log file created by software i can create a regex pattern by using a logstash filter combined with an IF condition - that's the reason for using logstash?!

But with this configuration it seems that filebeat is pushing the logs as plain text - I've thought that it is using templates and is working for most default log files - apache, syslog etc.

@timestamp: October 25th 2017, 13:05:45.611 offset: 96,200 @version: 1 beatname: testing_server beat.hostname: debian beat.version: 5.6.3 input_type: log host:debian
source: /var/log/daemon.log message: Oct 25 06:05:43 debian apachectl[913]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
type: logtags: beats_input_codec_plain_applied_id: AV9TNZefwb4CR4YbNQRC_type: log_index: filebeat-2017.10.25_score: -

Should be the bold part be parsed and not be within the message ?
What am I overlooking or am I completely wrong ?

PS: I will create another Topic later but maybe it is a very short answer.
Can i change the order of the tags in kibana? I prefer that the "message" is in a single/last line - which would make it easier to read?

Thanks!

[magnusbaeck (Magnus Bäck)]

I am aware that filebeat is able to push directly to elastic but if there is a untypical log file created by software i can create a regex pattern by using a logstash filter combined with an IF condition - that's the reason for using logstash?!

Yes, Logstash has richer features for parsing and processing logs.

Should be the bold part be parsed and not be within the message ?

Yes, the bold part should be chopped into separate fields. Getting that done is a typical use case for Logstash but recent Filebeat releases have a modules feature that might perform the same task. Unless you've enabled the right module Filebeat ships raw unparsed files.

Can i change the order of the tags in kibana?

I don't think so.

[sd2j]

Thanks for the information.

While searching for these modules I've found: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html
So it it not compatible with logstash, right?

Thanks

[magnusbaeck (Magnus Bäck)]

Compatible in what sense?

[sd2j]

"By default, Filebeat automatically loads the recommended template file, filebeat.template.json, if Elasticsearch output is enabled."
So the automatic parsing is only possible when filebeat -> elasticsearch is used and not filebeat ->logstash.

[magnusbaeck (Magnus Bäck)]

According to https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html you won't be able to use Filebeat modules together with Logstash.