What are the use cases for the file plugin v.s. logstash?

Elastic Stack Logstash
1

Hi,

I have looked at the file plugin. and am not quite sure what use cases is it meant for. Other than when sharing some sort of shared network storage, it seems as if this plugin can only ingest files from the local host, which is kind of limited.

Am I missing something or is logstash (e.g. https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) the solution to ingesting files from a remote host not sharing network storage?

Thanks!

2

I have looked at the file plugin. and am not quite sure what use cases is it meant for. Other than when sharing some sort of shared network storage, it seems as if this plugin can only ingest files from the local host, which is kind of limited.

How so? It's a plugin for reading files from locally mounted file systems. If that doesn't fit your use case there are dozens of other Logstash input plugins to choose from.

Am I missing something or is logstash (e.g. Elasticsearch output plugin | Logstash Reference [8.11] | Elastic) the solution to ingesting files from a remote host not sharing network storage?

The file plugin you're referring to is a plugin to Logstash. Comparing that plugin with Logstash itself doesn't make sense.

One normally installs a log shipping agent on all hosts that have logs whose contents you want to collect. The agent sends logs from the remove machines to the box where you want to collect all logs. Logstash, Filebeat, NXLog, and rsyslog are examples of programs capable of doing this.

3

If you want to compare it to something, look at filebeat - https://www.elastic.co/guide/en/beats/filebeat/current/index.html

4

Yes I've indeed mistaken the file plugin for an elasticsearch plugin rather
than a logstash one. Now it makes perfects sense.
Why the bother with logstash forwarder (
https://github.com/elastic/logstash-forwarder) and filebeat then?

5

Per the github page, LSF is deprecated.
Use filebeat :slight_smile:

6

Why the bother with logstash forwarder (
GitHub - elastic/logstash-forwarder: An experiment to cut logs in preparation for processing elsewhere. Replaced by Filebeat: https://github.com/elastic/beats/tree/master/filebeat) and filebeat then?

Because the disk and RAM overhead of Logstash can be prohibitive for small systems.

7

Thanks, I will try the beat! unlike logstash where you installed logstash and then used/added plugins, is it the case that beats are self-contained? I hope the file beat can well handle rotating-file logs and it is not too beta.

8

Perhaps I am not used to elastic documentation but it is hard to fine documentation / configuration instructions for the file beat.... and I find it baffling why would you forward beat information to logstash not directly to elastic as mentioned on the website....

Filebeat protects against this by quickly and reliably shipping logs to Logstash and Elasticsearch for centralized storage and analysis.

can you please advise on both accounts and about the former question on the file beat's overall status?

9

Filebeat does not support parsing logs and extracting fields, which is where Logstash comes in. In Elasticsearch 5.0, the concept of ingest nodes are being introduced, which will allow some processing within Elasticsearch and may allow certain architectures to be simplified by feeding data directly from Filebeat to ingest nodes.

10

unlike logstash where you installed logstash and then used/added plugins, is it the case that beats are self-contained?

Yes. A Beats-based binary is typically statically linked with an on-disk footprint of maybe a few tens of megabytes and a RAM footprint of a few megabytes.

I hope the file beat can well handle rotating-file logs and it is not too beta.

What gives you the impression that it's beta software?

Perhaps I am not used to elastic documentation but it is hard to fine documentation / configuration instructions for the file beat....

If I google "filebeat" the second hit is Filebeat quick start: installation and configuration | Filebeat Reference [8.11] | Elastic. Were you not able to find that documentation or was it hard to understand?

and I find it baffling why would you forward beat information to logstash not directly to elastic as mentioned on the website....

Logstash is able to process events in various ways that neither Filebeat or Elasticsearch is capable of. You typically don't send the raw logs to ES.

11

Thanks.
And still how to configure the file beat is very well conceiled to the beats newb. Can someone point me in the right direction?

12

Did you have a look at the documentation link I posted?

13

Yes, apologies, I didn't notice your message at the time. I really wonder what should the commands filebeat and filebeat-god be used for. filebeat seems to do nothing when I run it, it simply immediately returns without any output nor processes left running by it, as much as I can see.