Logs and no-index fields

Elastic Observability
1

Has anyone implemented a no-indexing strategy for their logs customers?

Meaning, I’d like to allow my customers to insert arbitrary data/structure, but not consume from the finite field count resource, so I want to map fields as index: false. The question is which fields? I'm wanting not to conflict with ECS.

I’m considering:

  • adding a tree where customers can dump info: ACME.noindex.*
  • specifying a prefix for fields that don’t get indexed: __ for example, as in __optype, service.node.__package_hash or __myDogsActivityPreferences.indoors.morning
2

Hey there @rsk0,

Both are viable options to get what you want.

If you have concerns over potential collisions with ECS fields, please check with the ECS documentation to get an overview of field names.

Thank you,

Coen Warmer

3

Thanks so much for your feedback.

I've been trying to make sure I understand ECS since we're using it as our foundation. The naming of my "no-index tree" option uses uppercase so as not to conflict, and the double underscore prefixing tries also to make use of guidelines and best practices while carving out something special that's not yet handled by ECS.

Now that I think about it, I should make sure to encourage proper naming of fields even if they're marked no-index by __. They may not be indexed now, but if they are switched to be indexed in the future we'll want the names conforming to ECS practices. So, __myDogsActivityPreferences.indoors.morning should be __my_dogs_activity_preferences.indoors.morning.

Do you think ECS might in the future designate space, somehow, for non-indexing data? Is that more a question for @ebeahan or @webmat?

[Edit: We are leaning towards using the __ designator for non-indexing fields.]