Select a distinct Set of values from a particular field aggregated by an other field

[aristotelis.sdrolias]

Hello all,

I am new on the field and I could use some help

Let's assume I have one index and one type and a list of documents within this type which include the following fields.

-- command_name
-- context_id
-- Other fields...

I can have a number of documents having the same context_id but different command name (relationship one to many). Example:

"context_id" : "1"
"command_name" Command_1

"context_id" : "1"
"command_name" Command_2

"context_id" : "1"
"command_name" Command_3

"context_id" : "2"
"command_name" Command_3

"context_id" : "2"
"command_name" Command_1

"context_id" : "2"
"command_name" Command_4

"context_id" : "3"
"command_name" Command_1

"context_id" : "3"
"command_name" Command_2

"context_id" : "3"
"command_name" Command_3

In this case I would like my query to return.

"
I have in total 2 sets. with the following population:
Set1 = (Command_1, Command_2, Command_3) with population 2 ( seen in 2 contexts (with id 1 and 3))
Set2 = (Command_1, Command_3, Command_4) with population 1 ( seen in 1 contexts (with id 2))
"

Am I able to query this information from Elastic and Visualise this information on Kibana?
Many thanks in advance for spending time on this.

Regards, Telis.

[jbudz (Jon Budzenski)]

Kibana will have a difficult time making inferences. Generally speaking it can group data together by common fields and then perform operations on those groups.

If Set1 and Set2 are the only definitions, you can bucket using the filter aggregation.
. Adding a sub bucket on context_id should then split each bucket by ids available. If the sets need to defined on the fly across documents it becomes more difficult. To get that case to visualize in Kibana I would suggest pre-processing the data set separately.

[aristotelis.sdrolias]

@jbudz

To get that case to visualize in Kibana I would suggest pre-processing the data set separately.

First of all, thanks a lot for your reply.
The sets are not predefined or known. We want to mine them...

When you say pre-processing, you mean inside elastic, in some way?
That might be for instance that on Elastic it is possible to create new document (or an object that sits on top of the source documents) based on information of other documents. Is that possible and straight forward? For instance on Oracle databases this could be accomplished with Views.
OR
you mean that the documents should be stored in a different way, meaning that pre-processing (ordered string aggregations in the above example) of data needs to happen before the data are entered in Elastic?