What the different beats to es and beats to logstash to es

Elastic Stack Beats
1

beats -> data -> es

beats -> logstash -> es

which good solution?

2

You would need to send the events to Logstash if you want to enrich the data. Otherwise you can just send the events directly to Elasticsearch.

For Topbeat and Packetbeat it is easier to just send the events directly to Elasticsearch. In the case of Filebeat, you would prefer to send the data via Logstash to parse the log lines and use grok patterns.

3

thanks reply

4

Hi Monica, thanks for your reply.

I'm new to logstash so I got a little bit confused by "enriching the data". So do you mean that logstash will automatically append some other fields (such as timestamp, ip address etc) to the events?

I'm also curious about how the mapping would look like when we use logstash and beats. As far as I can imagine, we can specify a template so that whenever an index is created, the predefined mapping can be automatically applied. Then how about if there is no such template? So the default analyzer will be applied right?

Thanks,

Xiaoting

5

Yes, in Logstash you can define grok patterns to parse and append other fields like timestamp. If you are using Filebeat, then we are encouraging our users to use Logstash (and not insert data into Elasticsearch) as Logstash is able to parse the log line and extract additional data that are important for you. Filebeat just forwards the log line to Logstash without any processing.

Each Beat has associated a mapping template available, but the template needs to be loaded in advance: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html#filebeat-template
If no template is loaded, then by default all the fields are set to "analyzed".

6

Thanks for your quick reply!

Just got one more question: let's consider this case, I'm monitoring a log of a server using filebeat. What would happen if either the server or the filebeat actually gets restarted? Will filebeat start from the first line of the log or it actually remembers the last processed line?

Thanks,

Xiaoting

7

Filebeats records where it got up to in the file in /var/lib/filebeat/registry. It is also configurable in the filebeat.yml.

If you want to reprocess files, you have to delete this file or modify the entry if you only want to reprocess one particular file.