Aggregate fields in a single field

[gius78 (Giuseppe)]

Hi all the forum from an elastic newbie.

I have different applications sending logs to logstash-kibana, each application has different fields in logs obviously, I'd like to aggregate fields containing same informations (for example name-surname of users)

I've tried the "copy" function:

PUT my_index/_mapping/user
{
       "properties": {
        "application1_name": {
          "type": "text",
          "copy_to": "universal_name" 
        },
        "application2_name": {
          "type": "text",
          "copy_to": "universal_name" 
        },
        "universal_name": {
          "type": "text"
        }
      }
    }
}

I have then refreshed my index but new "universal name" field does not show up.

First of all, I did not understood so much how my user defined indexes integrate with the "logstash*" auto generated indexes. I have always the feeling that documentation refers to knowledge I don't have.

Another thing, I'll probably have to aggregate other applications in the same fields, in the future. So I dunno if this solution is good, looks quite static.

thanks to all

[jbudz (Jon Budzenski)]

The universal_name field won't show up in _source, but it will return documents for terms queries of application1 and application2 names.

If you're storing these logs all in the same index, is a single application_name field an option? It's value can then be application1, application2, etc.

[gius78 (Giuseppe)]

Hi, thanks for reply.
When you say "universal_name field won't show up in _source" i am afraid of not knowing what is _source, I just wanted to have universal_name in the list of selectable fields in kibana, a field containing all usernames of the various applications.
Yes all applications are stored in the same logstash index, the fields have different names even if all share the same information (for example: user name and surname) probably I could name in the same way with a mutate in the logstash configuration, but I preferred another way.

Edit: Ok now I got what _source is, ok I don't expect universal_name being shown there, but i'd like to see in the list on the left.

[jbudz (Jon Budzenski)]

Unfortunately the list on the left is built from _source in the browser. To get it to show up we'll have to write to it ourselves or create a scripted field.

1. Like you mentioned - at ingestion time, using the logstash copy filter https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-copy
2. At search time in Kibana with a scripted field: from the index pattern's page, click on the scripted fields tab and add new:
   
   

if (!doc['application_name1'].empty) {
  return doc['application_name1'].value;
} else if (!doc['application_name2'].empty) {
  return doc['application_name2'].value;
} else {
  return null;
}

[gius78 (Giuseppe)]

Thanks again for the reply,

I created the scripted field, refreshed my index (dunno if necessary) but now zero documents are found in the discover, plus I get some kind of "shards error" in the top of the page (5 errors)