Hi,
I have an execution of a bunch of tests that run with multiple devices with multiple roles(DEVICE_1, DEVICE_2, etc..)
this means that for each test, I have for example 4 devices involved.
each device pushes its own data(and adds a role text field to identify), my goal is to take all the data lines from the 1st test(identified by test_id) and arrange them into one line(since all the data belong to test_id=0).
Up to now I just created a nested dictionary like this:
{INPUT:{ROLE:{FIELD:VALUE}}}
then, I pushed the dictionary(using the elastic python package) and this flattened the dictionary - this created a single line with all the data as I expected.
this is not a scalable solution for many devices since the role is part of the field.
is there any solution for it?
Hi @Aviel_Bashari,
Welcome back! Can you share your current index mapping to highlight the data structure you are pushing to Elasticsearch? Are you making use of nested fields or object types to represent the device information for each test?
sure!
I order the data inside the nested dictionary as I mentioned above, and then I know which field belongs to each device since the role is part of the field name.
@carly.richmond any advice, please? 
Can you share what the data looks like? I'm struggling to figure out which attributes refer to your device results versus to test.
assume that i have 2 participating devices named 'device1' and 'device2'.
each of them executes some operations and finally prints the same result.
in my dictionary I setting INPUT which is the parameters that I gave to the deviceX to run with, and OUTPUT which is the printed result after all operations.
cuureenlty I set inside the 2 keys(INPUT & OUTPUT) nested dictionary which contains the roles of the device for example 'CMP180'.
for example if I have an OUTPUT parameters called 'ANT' for 2 devices I will get 2 different fields:
OUTPUT_DEVICE1_ANT
OUTPUT_DEVICE2_ANT
And so on for all INPUT & OUTPUT fields.
this is not a scalable solution for my system since I have many devices in many executions.
attached screenshot with INOUT and OUTPUT for 1 device named 'CMP180'.
@carly.richmond as you can see in the attached screenshot:
I have a dictionary and inside I have INPUT and OUTPUT dictioinaries,inside I have the roles DUT/CMP and the data for each.
when I pushed this to ELK(using Elasticsearch package in python) I got the data as mention in my previous comment:
each field contains the device role - this is my big issue.
can you advice please
@carly.richmond any help, please?
Hi @Aviel_Bashari,
Thanks for reaching out. I've had a look at your data, and it really depends on what you want to search for/ aggregate when it comes to your test output. I did wonder if you could flatten the schema to generic terms and then make use of either nested or object arrays rather than hard-coding your field names as you have in your current schema.
Alternatively you could have a look at the flattened type and see if ingesting the test result json object as a single structure could help you avoid mapping explosion. But I would think about your queries and output first as they can impact the choice.
Hope that helps!
