Undocumented changes from Filebeat 6.7 to 7.0/7.1?

[LollerAgent]

We're working on upgrading from 6.6/6/7 to 7.1 and have noticed the following changes which we can't find mentions of in the release notes and/or documentation:

• When using add_docker_metadata, [container] used to be nested under [docker][container]. As of 7.x, it looks like [docker] is missing, and everything is under [container].

• Docker labels are "flat" now.

Pre-7.0:

	"docker": {
		"container": {
			"name": "container-name",
			"id": "058dcd11e4f719bd2514f6a7bf242a5a8b4233b8f111cfff159b83ba226b9cc6",
			"labels": {
				"com": {
					"amazonaws": {
						"ecs": {
							"task-definition-version": "1192",
							"container-name": "container-name",
							"task-definition-family": "task-family",
							"task-arn": "arn:aws:ecs:us-east-1:1234554545:task/b16c034f-daeb-4ba3-9176-ed239862d4b8",
							"cluster": "main"
						}

Post-7.x:

{
	"container": {
		"name": "container-name",
		"id": "058dcd11e4f719bd2514f6a7bf242a5a8b4233b8f111cfff159b83ba226b9cc6",
		"labels": {
			"com_amazonaws_ecs_task-arn": "arn:aws:ecs:us-east-1:12345678:task/b16c034f-daeb-4ba3-9176-ed239862d4b8",
			"com_amazonaws_ecs_cluster": "main",
			"com_amazonaws_ecs_task-definition-version": "1192",
			"com_amazonaws_ecs_container-name": "container-name",
			"com_amazonaws_ecs_task-definition-family": "task-family"
		},

Is there anyway to get the nested labels back?

• When using add_cloud_metadata, the metadata used to be nested under [meta][cloud], now it's just under [cloud].

Pre-7.x:

	"meta": {
		"cloud": {
			"availability_zone": "us-east-1d",
			"provider": "ec2",
			"instance_id": "i-jklfdslkfds",
			"machine_type": "m5.2xlarge",
			"region": "us-east-1"
		}
	},

Post-7.x:

	"cloud": {
		"instance": {
			"id": "i-fdsfdsfdsfdsfd"
		},
		"availability_zone": "us-east-1d",
		"provider": "aws",
		"region": "us-east-1",
		"machine": {
			"type": "m5.2xlarge"
		}
	}

Here is the configuration that we are using which has not changed since 6.x:

filebeat.inputs:
- type: docker
  containers.ids: '*'
  json.add_error_key: false
  json.message_key: message
  json.ignore_decoding_error: true
  processors:
  - add_docker_metadata: ~
  - add_cloud_metadata: ~

output.logstash:
  enabled: true
  hosts: ["${LOGSTASH_HOST}:10200"]
  ttl: 60s
  logging.to_files: false
  max_retries: -1

Can someone give me a hand here? Are these changes expected? If so, are they documented? Can I revert to the old nested version of docker labels?

[martinr_ubi (Martin R.)]

Hi,

This page from filebeat documentation:
https://www.elastic.co/guide/en/beats/filebeat/current/upgrading-filebeat.html

Links to everything you need: (release notes are links from breaking changes, so indirect link)
Breaking changes:
https://www.elastic.co/guide/en/beats/libbeat/7.1/breaking-changes-7.0.html
Release notes:
https://www.elastic.co/guide/en/beats/libbeat/7.1/release-notes.html
Upgrading docs:
https://www.elastic.co/guide/en/beats/libbeat/7.1/upgrading.html

Important->https://www.elastic.co/guide/en/beats/libbeat/7.1/upgrading-6-to-7.html#enable-ecs-compatibility

As far as reverting that, I'd say that you can't.
This is not an option it's a vision
https://www.elastic.co/guide/en/ecs/1.0/index.html

Elastic Common Schema (ECS): The Common Event Model for Elasticsearch

Think "unified logging", which is also a term used for this, shall I say, industry/best practice movement. It goes deep into the applications themselves for some proponents. Meaning all the app developers for a company try to emit in a common (structured) format. Usually logging via a common logging library or SDK.

"Why a different dashboard for Apache and Nginx... both are web servers!?"
"Why are those fields named differently if they both store the user agent string!?"
So I think* Elastic is just moving in that direction, which I see as a trend/best practice/industry evolution.
As most things though, my dad would say this isn't new, we're just still trying, harder.

CEF_White_Paper_20100722.pdf

I'm trying to say, maybe you shouldn't fight it, it's a good thing, at least that's the idea.

If you want to fight it?
I would suggest looking into either the ES ingest nodes processors or Logstash, if needed, to hack it into the old thing again. I think* filebeat can't do it with the processors it currently supports, I could be wrong though. But before doing that, make SURE you read this:
https://www.elastic.co/guide/en/beats/libbeat/7.1/upgrading-6-to-7.html#enable-ecs-compatibility

[LollerAgent]

Thanks. To be honest, this change was not conveyed very well. We normally check the release notes [1], and while SOME renames were mentioned in this document, the ones that matter to us were not.

This is not the first time we have been bitten by Filebeat release notes that aren't very clear.

[1] https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-7.0.0.html

[martinr_ubi (Martin R.)]

Hey @LollerAgent

1. I wasn’t specific enough in my answer, you raised multiple sub questions and I focused on only one; fields renames happening for ECS which I said could be hard to reverse and probably shouldn’t. I missed your question about the labels and the fact they are now dedotted by default. This is an option you’ll find documented on the processor page for add_docker_metadata, reversing is a simple matter of using the setting. They simply changed the default which is now true, yes it was in the changelog ;). The normal doc page for the processor is actually wrong though... the default is true now from 7.0. You want it false to get your dotted labels back. Just Do It (tm).
2. So sorry about misinforming you like that. I was too focused on the meta. and docker. fields rename.
3. I’d recommend you always read the release notes, the breaking changes and the upgrade docs for all the products of the stack your using. Even if you can’t remember everything when it’s a major version bump, you’ll remember it when tests fail or you bump against something while preparing your upgrades.
4. Sadly I can’t relate, they call me the Rain Man of the docs here, I quote docs or github issues or anything... every 4 sentences they say. But there are bad sides to this as well. If you caught it before impact, your doing good I’d say

[LollerAgent]

I appreciate the response! The problem with the labels isn't that they are dedotted by default now (for us, at least), it's that they are flattened like the example in my original post. In any case, we have made the necessary changes to our pipeline. I just hope these fields stay the same from now on.