# Undocumented changes from Filebeat 6.7 to 7.0/7.1? **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 4, 2019, 5:07pm UTC](https://discuss.elastic.co/t/undocumented-changes-from-filebeat-6-7-to-7-0-7-1/184233 "2019-06-04T17:07:07Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![LollerAgent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lolleragent/32/47486_2.png) [@LollerAgent](https://discuss.elastic.co/u/LollerAgent)\ **Post date:** [June 4, 2019, 5:07pm UTC](https://discuss.elastic.co/t/undocumented-changes-from-filebeat-6-7-to-7-0-7-1/184233/1 "2019-06-04T17:07:07Z")
We're working on upgrading from 6.6/6/7 to 7.1 and have noticed the following changes which we can't find mentions of in the release notes and/or documentation: - When using add\_docker\_metadata, [container] used to be nested under [docker][container]. As of 7.x, it looks like [docker] is missing, and everything is under [container]. - Docker labels are "flat" now. Pre-7.0: ``` "docker": { "container": { "name": "container-name", "id": "058dcd11e4f719bd2514f6a7bf242a5a8b4233b8f111cfff159b83ba226b9cc6", "labels": { "com": { "amazonaws": { "ecs": { "task-definition-version": "1192", "container-name": "container-name", "task-definition-family": "task-family", "task-arn": "arn:aws:ecs:us-east-1:1234554545:task/b16c034f-daeb-4ba3-9176-ed239862d4b8", "cluster": "main" } ``` Post-7.x: ``` { "container": { "name": "container-name", "id": "058dcd11e4f719bd2514f6a7bf242a5a8b4233b8f111cfff159b83ba226b9cc6", "labels": { "com_amazonaws_ecs_task-arn": "arn:aws:ecs:us-east-1:12345678:task/b16c034f-daeb-4ba3-9176-ed239862d4b8", "com_amazonaws_ecs_cluster": "main", "com_amazonaws_ecs_task-definition-version": "1192", "com_amazonaws_ecs_container-name": "container-name", "com_amazonaws_ecs_task-definition-family": "task-family" }, ``` Is there anyway to get the nested labels back? - When using add\_cloud\_metadata, the metadata used to be nested under [meta][cloud], now it's just under [cloud]. Pre-7.x: ``` "meta": { "cloud": { "availability_zone": "us-east-1d", "provider": "ec2", "instance_id": "i-jklfdslkfds", "machine_type": "m5.2xlarge", "region": "us-east-1" } }, ``` Post-7.x: ``` "cloud": { "instance": { "id": "i-fdsfdsfdsfdsfd" }, "availability_zone": "us-east-1d", "provider": "aws", "region": "us-east-1", "machine": { "type": "m5.2xlarge" } } ``` Here is the configuration that we are using which has not changed since 6.x: ``` filebeat.inputs: - type: docker containers.ids: '*' json.add_error_key: false json.message_key: message json.ignore_decoding_error: true processors: - add_docker_metadata: ~ - add_cloud_metadata: ~ output.logstash: enabled: true hosts: ["${LOGSTASH_HOST}:10200"] ttl: 60s logging.to_files: false max_retries: -1 ``` Can someone give me a hand here? Are these changes expected? If so, are they documented? Can I revert to the old nested version of docker labels? ---
**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\ **Post date:** [June 5, 2019, 2:55am UTC](https://discuss.elastic.co/t/undocumented-changes-from-filebeat-6-7-to-7-0-7-1/184233/2 "2019-06-05T02:55:29Z")
Hi, This page from filebeat documentation: [https://www.elastic.co/guide/en/beats/filebeat/current/upgrading-filebeat.html](https://www.elastic.co/guide/en/beats/filebeat/current/upgrading-filebeat.html) Links to everything you need: (release notes are links from breaking changes, so indirect link) Breaking changes: [https://www.elastic.co/guide/en/beats/libbeat/7.1/breaking-changes-7.0.html](https://www.elastic.co/guide/en/beats/libbeat/7.1/breaking-changes-7.0.html) Release notes: [https://www.elastic.co/guide/en/beats/libbeat/7.1/release-notes.html](https://www.elastic.co/guide/en/beats/libbeat/7.1/release-notes.html) Upgrading docs: [https://www.elastic.co/guide/en/beats/libbeat/7.1/upgrading.html](https://www.elastic.co/guide/en/beats/libbeat/7.1/upgrading.html) Important-\>[https://www.elastic.co/guide/en/beats/libbeat/7.1/upgrading-6-to-7.html#enable-ecs-compatibility](https://www.elastic.co/guide/en/beats/libbeat/7.1/upgrading-6-to-7.html#enable-ecs-compatibility) As far as reverting that, I'd say that you can't. This is not an option it's a vision 😄 [https://www.elastic.co/guide/en/ecs/1.0/index.html](https://www.elastic.co/guide/en/ecs/1.0/index.html) > **[Elastic Common Schema (ECS): The Common Event Model for Elasticsearch](https://www.elastic.co/blog/introducing-the-elastic-common-schema)** > > Introducing the Elastic Common Schema (ECS), a new specification that provides a consistent and customizable way to structure your data in Elasticsearch, helping machine learning jobs be applied more broadly, searches be crafted more narrowly, and... Think "unified logging", which is also a term used for this, shall I say, industry/best practice movement. It goes deep into the applications themselves for some proponents. Meaning all the app developers for a company try to emit in a common (structured) format. Usually logging via a common logging library or SDK. "Why a different dashboard for Apache and Nginx... both are web servers!?" "Why are those fields named differently if they both store the user agent string!?" So I think\* Elastic is just moving in that direction, which I see as a trend/best practice/industry evolution. As most things though, my dad would say this isn't new, we're just still trying, harder. 🙂 > **[CEF\_White\_Paper\_20100722.pdf](https://kc.mcafee.com/resources/sites/MCAFEE/content/live/CORP_KNOWLEDGEBASE/78000/KB78712/en_US/CEF_White_Paper_20100722.pdf)** I'm trying to say, maybe you shouldn't fight it, it's a good thing, at least that's the idea. If you want to fight it? I would suggest looking into either the [ES ingest nodes processors](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-processors.html) or [Logstash](https://www.elastic.co/guide/en/logstash/current/filter-plugins.html), if needed, to hack it into the old thing again. I think\* filebeat [can't do it](https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html) with the processors it currently supports, I could be wrong though. But before doing that, make SURE you read this: [https://www.elastic.co/guide/en/beats/libbeat/7.1/upgrading-6-to-7.html#enable-ecs-compatibility](https://www.elastic.co/guide/en/beats/libbeat/7.1/upgrading-6-to-7.html#enable-ecs-compatibility) ---
**Author:** ![LollerAgent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lolleragent/32/47486_2.png) [@LollerAgent](https://discuss.elastic.co/u/LollerAgent)\ **Post date:** [June 5, 2019, 2:00pm UTC](https://discuss.elastic.co/t/undocumented-changes-from-filebeat-6-7-to-7-0-7-1/184233/3 "2019-06-05T14:00:20Z")
Thanks. To be honest, this change was not conveyed very well. We normally check the release notes [1], and while SOME renames were mentioned in this document, the ones that matter to us were not. This is not the first time we have been bitten by Filebeat release notes that aren't very clear. [1] [https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-7.0.0.html](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-7.0.0.html) ---
**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\ **Post date:** [June 7, 2019, 9:07am UTC](https://discuss.elastic.co/t/undocumented-changes-from-filebeat-6-7-to-7-0-7-1/184233/4 "2019-06-07T09:07:28Z")
Hey @LollerAgent 1. I wasn’t specific enough in my answer, you raised multiple sub questions and I focused on only one; fields renames happening for ECS which I said could be hard to reverse and probably shouldn’t. I missed your question about the labels and the fact they are now dedotted by default. This is an option you’ll find documented on the processor page for add\_docker\_metadata, reversing is a simple matter of using the setting. They simply changed the default which is now true, yes it was in the changelog ;). The normal doc page for the processor is actually wrong though... the default is true now from 7.0. You want it false to get your dotted labels back. Just Do It (tm). 2. So sorry about misinforming you like that. I was too focused on the meta. and docker. fields rename. 3. I’d recommend you always read the release notes, the breaking changes and the upgrade docs for all the products of the stack your using. Even if you can’t remember everything when it’s a major version bump, you’ll remember it when tests fail or you bump against something while preparing your upgrades. 4. Sadly I can’t relate, they call me the [Rain Man](https://en.m.wikipedia.org/wiki/Rain_Man) of the docs here, I quote docs or github issues or anything... every 4 sentences they say. 🤓 But there are bad sides to this as well. If you caught it before impact, your doing good I’d say 🧪 ---
**Author:** ![LollerAgent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lolleragent/32/47486_2.png) [@LollerAgent](https://discuss.elastic.co/u/LollerAgent)\ **Post date:** [June 7, 2019, 1:08pm UTC](https://discuss.elastic.co/t/undocumented-changes-from-filebeat-6-7-to-7-0-7-1/184233/5 "2019-06-07T13:08:00Z")
I appreciate the response! The problem with the labels isn't that they are dedotted by default now (for us, at least), it's that they are flattened like the example in my original post. In any case, we have made the necessary changes to our pipeline. I just hope these fields stay the same from now on. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2019, 1:08pm UTC](https://discuss.elastic.co/t/undocumented-changes-from-filebeat-6-7-to-7-0-7-1/184233/6 "2019-07-05T13:08:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.