When to use "Object" field datatype vs flat fieldnames

Elastic Stack Elasticsearch
3

Hi @nverrill,

There's two things I see in your question

  1. Whether to nest or not to nest
  2. Whether dots should represent nesting or literal dots in your key names

You didn't ask #2, but you may stumble upon it as well.

To answer #1 directly -- your question -- nested objects are slightly easier to handle programmatically. E.g. you can delete or copy source to affect all of its subkeys at once, instead of looping over them, looking for "source_" keys. It also makes a clearer delineation between what's a section and what's simply two words joined together to represent a concept. Consider source_top_level_domain vs source.top_level_domain. The latter makes it clear what's a section name -- "source" -- and that there's a key named "top_level_domain" in there.

To answer #2 -- why nesting instead of literal dots: that's actually a decision that predates ECS by a long time. I think moving away from literal dots is an Elastic Stack 5.0 decision, iirc. As such, Logstash supported both and still does. Elasticsearch ingest pipelines only support "dedotting" dotted keys (or replacing them with nesting), but otherwise doesn't support dotted keys. I think the same is true of Beats processors as well. So for that part, the future is with "." means nesting, not literal dot.

Finally, in any case where you're adding custom fields, you're more than welcome to name them however you want; you can solely use underscore for your custom fields, if you want :+1:

But the ECS fields are nested objects, and the dots are not literal, they represent nesting. If you want to follow the schema, that's how the keys are named :slight_smile:

You can read some more at https://www.elastic.co/guide/en/ecs/current/ecs-faq.html#dot-notation and https://www.elastic.co/guide/en/ecs/current/ecs-guidelines.html#_guidelines_for_field_names.

1 Like