Kibana data table visualization and list of arrays question

Elastic Stack Kibana
1

Hello!

Our antispam gateway is pushing metadata to our elasticsearch cluster.

A pushed doc json looks like this:

{
   "rspamd_meta":{
      "rspamd_server":"out-mta01",
      "message_id":"cf1a3fdd58f966a299e1b820d67ee9f5@email.bi",
      "header_to":[
         "email@email.com"
      ],
      "asn":{
         "country":"CH",
         "asn":"49xx7",
         "ipnet":"94.xx.xx.0/20"
      },
      "webmail":false,
      "ip":"94.103.96.175",
      "header_from":[
         "\"Bienvenue sur le site\" <some.sender@email.bi>"
      ],
      "from":"someuser@web15.ourservers.com",
      "rcpt":[
         "email@email.com"
      ],
      "score":100,
      "user":"someuser",
      "direction":"Outbound",
      "hostname":"web15.ourservers.com",
      "symbols":[
         {
            "group":"ungrouped",
            "groups":[

            ],
            "options":[
               "email@email.com"
            ],
            "name":"BLACKLISTED_RCPT",
            "score":100
         },
         {
            "group":"ungrouped",
            "options":[
               "asn:49457, ipnet:94.x.x.0/20, country:CH"
            ],
            "name":"ASN",
            "score":0
         }
      ],
      "action":"reject",
      "is_local":false,
      "header_date":[
         "Tue, 31 Mar 2020 18:12:25 +0200"
      ],
      "header_subject":[
         "Bienvenue sur le site: Hey, wie ist deine Woche?"
      ],
      "qid":"6E452386-3BBF-4267-A1B6-08FB0FCC373C.1"
   },
   "@timestamp":"1585671145930.2"
}

So rspamd_meta.symbolsis pushed as an array of list (or the reverse i'm not really sure here)

The index pattern for the documents looks like:

kibanapatterns
kibanapatterns944×215 12.2 KB

In discover the result is:

kibanalistarrays
kibanalistarrays855×468 44.3 KB

What I am trying to do is, in our data table visualization, add a column that reports the symbols.name with the highest symbols.score.

I tried to do this adding a split row bucket on rspamd_meta.symbols.name limited to 1 result and sorted on custom metric based on rspamd_meta.symbols.score descending, but it's not giving the expected results.

Am I doing it wrong? Any idea how I could achieve this ?

Thanks a lot for any help!

2

Update: After a bit of reading, I tried to set the field type to "flattened" and reindexing all indices, but it's even worse. Kibana now shows the field as type "unknown".
I guess it's not the right path to follow to achieve what I'm trying to do :slight_smile:

3

Kibana does not supported nested for flattened fields. It may be best to break the array into individual documents