Filebeat types elastic 6

Elastic Stack Beats
1

Hi,
My setup is:
elasticsearch cluster < logstash_host < ( filebeat_host, filebeat_host2)

Is there a need / benefit to defining log types in elastic 6?
https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html

What's the advantage of using the defined module e.g. apache2 module over the standard log module with paths: defined?

Since I'm using logstash, is there an easy way to install all indexes required at point of use e.g. if no apache hosts are configured, don't install the apache indexes, or is this not possible? Is there a downside to installing these generic indexes, even if they're not used?

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html

2
  • Types: What do you mean by log types?
  • Apache2 module: If you use the module, an ingest processor can be loaded that does the processing of the logs (grok) for you. But that works best when sending the data directly to Elasticsearch.
  • To load all the templates for beats you can take one filebeat instances and run ./filebeat setup. There is only 1 template for all modules and only 1 daily index is normally used. What you mean by generic indices?
3

Types - I think I've misread this and it's the meta _type which has been removed

Apache2: so if I use the default it'll do something similar to:
filter {
if [type] == "apache" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}

Without having the specify this in my logstash conf?

3 - again I may be misunderstanding this. The link says if you're using ELK then you can't load the templates and instead need to be done manually (https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-template.html#load-template-manually). Since it's filebeat they will always be filebeat* indexes, as opposed to:
apache-log*
system-log*
correct?

If I was to install the filebeat* index into elastic via logstash, how can I add any subsequent indexes automatically e.g. say if I add another beat (e.g. packetbeat)? Is there a downside to creating packetbeat* anotherbeat* templates. even if they're never used?

I'm beginning to think it might be easier to use EK as opposed to ELK - any thought / opinions on this?

4

  1. Yes, but you will not need LS in this case as data goes directly to ES.

  2. You can use specific indices. You can specify them in the output also based on criterias if you want: https://www.elastic.co/guide/en/elastic-stack-overview/6.3/get-started-elastic-stack.html#logstash-setup The important part is that the templates match the index patterns. If you use filebeat-* it just works. If not, you have to do manual modifications.

This should also answer your question about packetbeat: You can use the meta information on LS into which index it's sent.

If you just have apache logs, you probably don't need Logstash. Logstash comes into play when you need enhanced processing, local queue, multiple outputs etc. So it really depends on your use case.

5

So you can't use the modules with a logstash endpoint?

6

You can but it's not an out of the box experience and the processing would still happen in Elasticsearch ingest.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.