Visualization with conditional aggregation

#1

Hi there,

Let's say I define the index like below

PUT my_index/_doc/1
{
  "user": [
    {
      "first": "John",
      "last": "Frank"
    },
    {
      "first": "Hero",
      "last": "Tim"
    }
  ]
}

PUT my_index/_doc/2
{
  "user": [
    {
      "firstName": "John",
      "lastName": "Term"
    },
    {
      "firstName": "David",
      "lastName": "Gayle"
    }
  ]
}

If I want to list all the user surname whose firstName is John in Data Table visualization, is there a way to accomplish this?

#2

Kibana does not support nested objects yet, https://www.elastic.co/guide/en/kibana/current/nested-objects.html.

I would recommend indexing your data in a flatter structure like below.

PUT my_index/_doc/1
{
  "user": {
      "first": "John",
      "last": "Frank"
    }
}

PUT my_index/_doc/2
{
  "user": {
      "first": "Hero",
      "last": "Tim"
    }
}
#3

Thanks for the reply, Nathan_Reese

Currently, these logs come from other services which are in nested object. Hence, If I want to modify the log to be flatter structure like you mentioned (for example, create a new log for each user in order to be aggregatable), does Elasticsearch support it?

#4

How are your logs getting ingested into Elasticsearch?

Try using ingest node to pre-process documents before indexing

#5

Yes, I currently use the ingest node to manipulate logs. The log structure before ingested is liked I mentioned above

"user": [
    {
      "first": "John",
      "last": "Frank"
    },
    {
      "first": "Hero",
      "last": "Tim"
    }
  ]

in a single document
So, can ingest node do like creating new documents from receiving logs
for ex. spliting user array into two new documents as you mentioned

#6

yes, you should be able to manipulate the document with ingest node, see documentation for details

#7

I have read it. I found only Set Processor that's more likely to manipulate logs, but it's not what I want. I want to create new document while the data is processing in ingest node. Could you please point out how to do so?

#8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.