Best Practices / Recommendations for Custom Objects

Elastic Observability Logs
,
1

We are evaluating ECS as a common transport schema for our events, logging, and messaging needs.

Tags are not enough as we want to have sharable serialized compound objects.

We initially started with the .NET NLog mapping implementation. Custom data was placed in a "metadata" node. However, it serializes as "_metadata" and gets ignored in newer versions of Elastic/Filebeat. This questions its viability.

Should we just rename "_metadata" to "metadata?" Would we be impacted and conflict with any future plans for this node? Should we avoid it?

Should we just create a new top-level node for our purposes, such as "customData?" Should we reuse and extend another already existing node?

Again I am just looking for the best practices for a solid future-proof implementation.

2

Hi @Jeff_Stevens,

those are interesting questions. In my mind it roughly comes down to two aspects: the field name(s) and the field type(s).

As for the field names, the custom fields chapter in the ECS docs recommend the following:

  • If your custom data is mostly key-value pairs with keyword semantics, using the labels ECS field sounds reasonable.

  • If the structure is completely arbitrary, opening up a custom namespace for your organization or product sounds like the safest way. It would have the lowest risk of a naming collision. The ECS custom field guidelines suggest capitalizing the field name to prevent any collision with future ECS field names.

For the field type(s) it mostly comes down to the queries you expect to run against the metadata:

  • If you only expect to retrieve the data after looking up the document using a different set of indexed fields, setting the type to object and also setting "enabled": false would be most efficient and give you full freedom to store any JSON-compatible values. You would also retain the option of making the values searchable at runtime via runtime fields, albeit at a performance cost.

  • If you expect to have to query for keys and values, the keyword-like query semantics of the flattened field type might offer a good compromise between flexibility and searchability.

Does that help you with your deliberation?

3

Hi @Jeff_Stevens, let me see if I can answer your questions.

This is a bug that has been addressed in

and is in the new 1.5.3 release on Nuget:

"_metadata" was originally chosen in order to not conflict with any potential future ECS "metadata" field that might be introduced.

The intention with the Elastic.CommonSchema project is to provide a base implementation of ECS types in .NET. Understandably, there may be situations where the ECS types or an ECS logging integration may not be flexible enough;

Extending ECS types

In simple cases, key/values can be added to the Base.Metadata property and they'll be serialized under "metadata".

In more complex cases, it is possible to derive from any of the ECS types to provide additional properties. An example can be found in the BenchmarkDotNetExporter, where Base and other ECS types are derived from to add benchmark results for indexing into Elasticsearch

Another example is the LogEvent used by the Elasticsearch.Extensions.Logging project (which will be released in the next minor), which derives from Base to add a MessageTemplate and Scopes properties

Extending ECS logging integrations

It is also possible to extend ECS logging integrations by deriving from EcsTextFormatter or EcsLayout to capture additional values or work with your own ECS derived types.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.