I have a system that writes it's logs to elastic for a while now.
We have a feature in the system that allows the user to enter tags.
The tags can have any name and any value (both are strings).
We want to start and log these tags so they can be visualized and searched.
But, we encountered a problem while trying to do so.
The first naive approach we had, was to index any new tag in its own field.
This approach has its issues of course as we passed the maximum allowed fields.
We thought of upping it up but we soon passed our new limit. We currently have more than 4000 unique tags.
So, our instinct was to redesign our logs and write them as objects like this:
This solves the issue of having too many fields. But, it has problems with search. As I can get this document when looking for one with "key" as tag_a and a "val" with val_b.
This is a problem for us so we naturally decided to look at nested documents which will solve our problems.
Unfortunately, Kibana does not support nested documents in Discover or Visualizations.
Kibana is our main UI and we don't wish to develop something else to enable the search of our nested data.
We'd like to hear of any other ways to index the data that we might have overlooked to enable our use case.
Thank you very much