Mapper-Size Query

Elastic Stack Kibana
1

Hi,
Could someone please comment on the discussion in this thread:

Regards,
David

2

Hi David,

I've read through your post in #elasticsearch and have been trying to reproduce your scenario. Unfortunately I'm not having much success. Here's what I'm observing; maybe you can tell me what you are observing that's different:

I installed the mapper-size plugin in Elasticsearch 2.2.0 and creating an index with the following mapping:

PUT foo
{
  "mappings": {
    "bar": {
      "_size": {
        "enabled": true
      }
    }
  }
}

Then I indexed a couple of documents into it, of varying sizes.

When I create an index pattern for this index in Kibana, I do not see the _size field in the list displayed by Kibana:

Screen Shot 2016-06-28 at 7.45.38 AM.png2600×1286 320 KB

Given that, I cannot use the _size field in any visualizations for performing aggregations on it.

3

Hi,
Silly question but did you restart your elasticsearch to enable the plugin?

Regards,
David

4

Hi! Yes, I stopped Elasticsearch to install the plugin, installed the plugin, and then started up Elasticsearch again.

5

When you get the mapping for the index you have "_size" defined for what do you see?

6

The initial mapping, right after I create the index (as described in Mapper-Size Query) looks like this:

$ curl 'http://localhost:9200/foo/_mapping?pretty'
{
  "foo" : {
    "mappings" : {
      "bar" : {
        "_size" : {
          "enabled" : true
        }
      }
    }
  }
}

After I indexed a couple of documents with field p1 containing strings of varying sizes, my mapping looks like this:

$ curl 'http://localhost:9200/foo/_mapping?pretty'
{
  "foo" : {
    "mappings" : {
      "bar" : {
        "_size" : {
          "enabled" : true
        },
        "properties" : {
          "p1" : {
            "type" : "string"
          }
        }
      }
    }
  }
}
7

Have you updated the metafields definition in the advanced settings?

1 Like
8

Is this no longer being looked at?

9

Hi David,

Apologies! I rely on getting email notifications whenever there are updates to a topic but that seems to have broken somehow because I never got notifications about your latest two posts.

Anyway, I updated metaFields in Advanced Settings and added _size to that list. After that, if I refresh the index pattern definition for my index, I see _size in the list but with type as unknown:

Screen Shot 2016-07-14 at 8.06.25 AM.png2794×1916 399 KB

As Kibana doesn't know the type of this field, it doesn't know which aggregations can use it. Therefore it doesn't show it in the visualization editor.

Next I'm looking into why Kibana is unable to figure out the type of this field. I'm guessing this information is not being returned from Elasticsearch but let me confirm that.

1 Like
10

Hi @dawiro,

I finally got some time to dig into the Kibana code and figure out what's going on here.

The summary is that Elasticsearch does not report the datatype of the _size field to Kibana so Kibana sets it to "unknown". Without knowing a field's type Kibana cannot determine which aggregations can use this field so Kibana makes the field unavailable in the visualization editor.

Read on for more specific details on what's going on in Kibana:

In order to generate the list of fields for an index pattern, Kibana must request the mappings for all fields from Elasticsearch. To do this it makes a REST API call to Elasticsearch like this: GET {host}:{port}/{index-pattern}/_mapping/field/*. This call returns a response that looks something like this:

{
  "{index-name}" : {
    "mappings" : {
      "{type}" : {
        ... mappings for some fields ...,
        "_size" : {
          "full_name" : "_size",
          "mapping" : {
            "_size" : {
              "enabled" : true
            }
          }
        },
        ... mappings for remaining fields ...
      }
    }
  }
}

In this response Kibana looks for a type property in each field's mapping. As you can see in the example response above the _size field's mapping does not contain a type property. Since Kibana does not find the type property it sets the type of the field to "unknown".

If you'd like, please create an enhancement request issue in the Elasticsearch Github repository asking the mapper size plugin to report its type in the API response above.

11

Thanks! it's good that you've been able to reproduce my findings...

12

Hi There,
this is very useful information to be visualized also for me.
I was investigating and realized one work-around for this - you can create Scripted field and then use it:

  • create scripted field, name it whatever you want (ie. _doc_size)
  • make this numeric field
  • script: doc['_size'].value

that's that!
Hopefully this will help someone.

Marcel

2 Likes
13

It helped a lot, thank you Marcel.

screencapture-localhost-5601-app-kibana-1496318425067.png1920×1315 267 KB