Handling different flows in data streams

Elastic Stack Elasticsearch
,
1

Hello,

I have been using data streams for my business flow logs, each flow has the different log structure. I have couple of solutions in mind:

  1. Having same structure for each flow and keep stringified version of log object in particular key. For example:
{
                tag: '--ELK--',
                function: '<function name>',
                options: <stringified object>
}

this will inefficient in searching

  1. Having same structure for each flow and keep nested object of log object in particular key. For example:
{
                tag: '--ELK--',
                function: '<function name>',
                options: <JSON object>
}

this will add complexity in writing in the indices as lucene index does not have nested object support.

  1. Is there a way to control the index creation in the data stream in such a way that index could be created as per flow for example: if I have login, signup, and logout flows, in my logs-datastream if I could create different index in the data stream for each flow so that each index would have it's own structure.

Kindly suggest. TIA

1 Like
2

Another thing I can add over here is parsing the data as fields and values,

{
   "key": "property_name",
   "value": "property_value"
}