Schema differences 6.6.0 vs 7.x

[digitalron (Ronald Tumulak)]

I have started looking at 7.0.0-beta1 and have noticed that the schema has changed significantly from 6.6.0. We are rolling out our Elastic APM stack by June and are wondering if we should go ahead with 6.6.0 or wait until 7.x becomes GA.

We have a few ingest pipelines that will be impacted when the context.* fields (and others) are removed/changed from 7.x. Any advice for this?

Also, is there a reference table where we could compare all changes where there is a recommendation on how to plan to migrate from pre 7.x fields to 7.x fields?

Thanks!

[gil (Gil Raphaelli)]

Hi again @digitalron, excellent questions - I'll try to address them in reverse order.

There are indeed many fields moving around from 6.x to 7.0 in an effort to align the APM schema with the Elastic Common Schema. The summary is captured here with additional documentation in the works ahead of the 7.0 GA release.

We are providing a migration assistant in Kibana for reindexing documents from 6.x to 7.0 format - you can experiment with this using that latest preview release of 7.0.0, currently beta1. It's emphasized in the assistant but please be sure to backup any valuable data before proceeding, the assistant will drop the original index following migration.

If the migration assistant is sufficient for your needs I don't think you need to wait until 7.0 to deploy - go out with 6.6 first and then migrate docs as needed. There is no requirement to upgrade Java agents between 6.6 and 7.0, only the apm-server component.

Can you provide more details around the ingest pipelines you are utilizing? I expect you'll need map them to new fields using the summary provided.

[digitalron (Ronald Tumulak)]

Hi Gil, thank you very much for these information, especially the links. Those are mighty useful indeed!

I understand and fully subscribe to these changes and see the benefits of having the Common Schema: it actually helps our single observability stack project moving forward.

From our side, we created ingest rules for user_agent and geoip, and as an example, mapped context.user.ip and context.system.ip to normalise on our custom user.geoip.* fields with a definition for geopoint and other related data.

With the coming change, it means either our current ingest and fields definitions have to change (easier but may not conform to ECS intentions), or consumers of our custom user.geoip.* fields have to be modified (visualisations, Query DSLs consumed by external systems, Timelion, and aggregations). No big deal really; we'll just have to make sure we align with the core principles of the ECS.

-- edit (additional input) --

One other thing, we were looking to use some 3rd party Kibana plugins but are no longer sure if it would be a good idea to proceed with that due to the impending 7.x release.