@axw certainly! First a disclaimer: This is just my personal opinion! I perfectly understand that there is probably many aspects under the hood that I'm not aware of
My immediate problem is that the name of my attributes defined in my system change when entering Elastic. This requires mental mapping with Kibana usage and technical if we are to query the API directly. I can see that allowing attribute names with dots but keeping the structure flattened would solve some of this, at least for the Kibana usage. In that case, UI improvements would be greatly appreciated as I really like being able to dot through the nested structure in the Kibana.
I understood the potential mapping conflict you are trying to solve. However, I think it seems like a lot of work to hide the design problem the user is having the their defined structure. If you define both x
and x.y
as a string you have not defined x
clearly enough - it has multiple uses. To me, this is not much different than defining x
as string in one place and as integer in another place. This would also result in a mapping conflict as far as I understand (and rightfully so).
I ran into a bit of the same problem when trying to get my transaction to be marked as messaging instead of known. I had defined the Otel attribute messaging.destination.name
in accordance with the latest Otel semantic conventions (1.21), however digging through apm-data
showed it was still using messaging.destination
(1.5).
The solution you have currently described is also in direct contrast to the guidelines of your own ECS Guidelines which I referred to previously.
Nest fields inside a field set with dots
The document structure should be nested JSON objects. If you use Beats or Logstash, the nesting of JSON objects is done for you automatically. If you’re ingesting to Elasticsearch using the API, your fields must be nested objects, not strings containing dots.
With the contribution of ECS to OpenTelemetry (which I applaud!), I think this also applies to an OpenTelemetry integration, at least in the long-term.
I assume that non-custom fields are handled hierarchical, so I think it would be better to embrace the hierarchical structure for all data. I think we all face the design issue from time to time of having defined something as primitive but later finding out that a complex type would be better suited. Instead of trying to hide this I think it would be awesome if you could help with changing a primitive type to a complex type.
One way I could see this implemented could be to define a convention for a default value for a complex type. Ex. {x: "qwerty"}
could re-indexed to {x: { _value: "qwerty" }}
. The query with a primitive type on a complex attribute could transparently map to this nested convention attribute. I find this similar to how ex. HTML can nest both dom-elements and text at the same time.
I understand that people might have widely different use cases, but personally I would much rather have a "your on your own"-option for APM and not have APM assume that I make mapping conflicts.
I hope you find this helpful, and not too much wall of text