Beats and Composable Templates

Elastic Stack Beats
, ,
1

Hi,

I am in the process of upgrading our Elastic Stack instance from 7.5 -> 8, and am going through all of the migration tasks.

It looks like migration to component templates is going to be a pain.
Currently, I am using template inheritance/merging, making use of the order field.

For our logs where we have domain specific fields not captured in the ECS, we have dedicated indices which inherit from filebeat in the first instance, and we map as much as possible to the ECS fields. Then our other higher-order number templates define all the other fields we require.

Additionally, I use dynamic_templates to override the log.original field to not truncate the log message at 1024 characters. This is required when parsing the message fails, we log the whole original message to a separate index from which we can look at the message and attempt to re-index the message after fixing the filters.

Having tried setting up composable template using filebeat 8.7.1, I have noticed that it has no components, it is just a monolithic composable template, which means we can inherit nothing as I understand it.

Here is the command I used:

.\filebeat.exe setup --index-management -E output.logstash.enabled=false -E output.elasticsearch.hosts=["localhost:9200"] -E output.elasticsearch.username="#####" -E output.elasticsearch.pa
ssword="#####" -E setup.template.type="component"

Here is the index management page in Kibana showing the filebeat template

image
image1168×465 35.5 KB

I do not want to have to maintain our own version of the filebeat template, it defeats the purpose of having composable templates, and the original "legacy" templates did allow this reuse.

For the time being I will have to continue with the legacy templates until this is resolved.

Is there a plan for the beats to create their templates in a more component template style allowing reuse?

Thanks in advance.

2 Likes
2

Love this question we have been having serious problems with this for months. Elastic clearly didn't think this one through, we warned support and our account manager of this the moment component templates were communicated, we never got a real answer, they even acknowledged it is a problem, but no solution in sight afaik. It is currently very hard to override a builtin template without risking problems after an update. This problem has seriously increased the time needed to implement our update procedure, as the steps required are not automatable. We need to test the templates first on QA, see what's changed and then implement the custom overrides.

This issues has multiple sub issues by they way. Actually fields can be added with a custom component template, but this still implies the index template itself needs to be reconfigured to contain the custom template. Also, existing fields can not be overriden with for example a lowercase normalizer.

1 Like
3

I currently have a case open with them for this, it's still being looked at.

I can understand some of the sub-issues you mention, like overriding fields which i do with the log.original field.

If the beats team created component templates which also broke down their mappings into components rougly corresponding to their modules, it would also reduce the number of fields contained in the mappings.

For instance, in my case, I am reading from log files using the log module. I really only need the fields from Filebeat/ECS

  • agent.*
  • ecs.*
  • event.*
  • host.*
  • log.*
  • process.*

All the fields are populated when using relevant modules like Apache, Azure, AWS, etc. are never used or populated for our systems.

My own templates are layered over the top for our domain-specific data which has no corresponding fields in the ECS.

The only thing I had to do to the official template to be able to inehrit it for our custom indices was add our own index pattern to the index_patterns array in the filebeat template, example below:

{
  "index_patterns": [
    "filebeat-7.17.10-*",
    "custom-index-*"
  ],

This was a simple enough operation.

I am hoping that the beats team will catch up with the spirit of composable templates and create
them along the lines I have mentioned.

This would greately simplify things and allow flexible reuse without needing to create my own copy of their template and manage it our codebase, and manage updates to it.

1 Like
4

I think that the focus is now on the Elastic Agent and its integrations, there are some current issues on github that tackle the many problems of customization with the Elastic Agent (templates, mappings, pipelines and lifecycle policies), but I would not expect to see this reflected on the individual beats.

1 Like
5

Okay, I have not taken a look yet at the elastic agents.

I will have significant challenges in my environment with connectivity and deployment.
I will need to be able to manually load the agents assets (templates, index patterns, dashboards and visualisations, etc). Currently we do this by dumping them using the export command or copying the contents of the kibana folder. All of this is automated.

I am not certain that I will be permitted to use fleet to manage the agents, and they will never have direct connectivity to the elasticsearch cluster.

We currently make heavy use of logstash for handling message processing.

I will have to look into this more.

1 Like
6

You can install the elast agent in a standalone mode, but the documentation is not very good for this case.

Also, you will have probably the same issues you have now with filebeat and maybe some more with the Elastic Agent as it has some limitations even if behind the scenes the Elastic Agent is running a Filebeat instance for example.

Another thing is, if you are not going to use the integrations, then it does not make much sense to use the Elastic Agent now.

My experience with the Elastic Agent is that while it makes easier to add and parse some kind of logs, it makes everything else way, way harder, manage things created by the Elastic Agent can be a nightmare if you need to customize anything.

Last year We made some tests with the Elastic Agent Integrations to try replace some Logstash pipelines, but on our use case we need a lot of custom mappings, processors and different retentions and those things does not go well with the Elastic Agent, anything custom will need a lot more of work, for example, this issue that I opened explains some of the problems we faced.

In the end we just added a couple of simple integrations (basically audit logs from SaaS) and stopped there, everything else stayed on Logstash as we were spending more time trying to manage things with the Elastic Agent than if we built a custom Logstash pipeline.

Yeah, I also make heavy use of Logstash, which makes the Elastic Agent redundant, for example, if I need Logstash to do some processing on the log messages for a Palo Alto device, then it makes no sense to use the Palo Alto integration of the Elastic Agent.

And some things are only possible or work better with Logstash (mainly enriching of data from local and external sources).

I'm curious on why you need the filebeat template, I think that I had a similar use case as yours and was pretty easy to migrate to component templates, but I use custom indices names, never used the filebeat-* indices.

Do you use filebeat-* indices? Wouldn't migrate to custom indices/data stream make it easier for you to manage things?

7

The sad thing is that overriding builtin templates worked perfectly with the legacy templates for years..... We've never had such a long update procedure since Elastic 8.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.