Is the winlogbeat 6 template backwards compatible to 5.x?

[kwslavens (Kelly Slavens)]

Is it possible to feed winlogbeat 5.x agents to a index using a 6.0.0 template? Is the template backwards compatible?

Here is the problem, with the way the new winlogbeat agents break out the templates into separate names.

We've got 11K of agents running 5.4.2. Our current template for winlogbeat-* is version 5.6.4.
When I imported the new 6.0.0 template it did not replace the current "winlogbeat" template. It instead created a new template named winlogbeat-6.0.0 which overlaps with the current 5.6.4 winlogbeat template.

  "winlogbeat-6.0.0": {
    "order": 1,
    "index_patterns": [
      "winlogbeat-6.0.0-*"

and

"winlogbeat": {
"order": 0,
"index_patterns": [
"winlogbeat-*"

These two patterns now overlap.

In addition our configuration does not currently divide agent traffic by version. We run through a kafka queue that uses's a single topic for all winlogbeat agents.

Are we going to have to split the traffic by version and make seperate indexes for every beats version?

[ruflin]

We introduced indices and templates for each version to ensure better compatibility in the future. One of the main reasons here is Metricbeat where new fields are added pretty frequently. Winlogbeat should be affected by this much less.

AFAIK there are very little change in Winlogbeat fields from 5.x to 6.0. So I think 6.0 should also work with the older versions but I would definitively encourage you to try it out first to see if you get some errors. @andrewkroh might know more.

Coming back to your setup with Kafka: I would encourage you to send the data for each version from Winlogbeat to it's own index for all 6.x release to make sure in case we have changes to the template, you also get them directly. If you send all data to one index, make sure that the newest template from the 6.x release is always applied as during a major version the template should always be backward compatible with data coming in from previous minor versions.

[kwslavens (Kelly Slavens)]

Thanks, We've broken out the beats agents into {beatype}-{beatversion} -datetime format to cover this new change.

One thing this brings to mind as a concern is that now I'm subject to unplanned shard bloat for the various revisions of the agents. In a large organization with 10,000+ agents its very unlikely to get all of them upgraded reliable. Right now for instance I see 4 different versions of winlogbeat. Where this use to total only 10 shards per/day, now this will jump to 40 shards per/day.

This may prove to be a problem for us long term as we hold data for 13 months.

[ruflin]

Yes, that is a valid concern. Some options here are:

• Recheck how many shards you need per index. This can now be configured through the beats config.
• Shrink the indices which are done with writing to 1 shard: https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-shrink-index.html
• Use rollover instead of daily indices as often daily indices are either too small or too large: https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html