"docs_value" doesn't seem to take affect

Elastic Stack Elasticsearch
1

We use ES 1.4.4 . We configured some fields to be "docs_value", however we still see that the field data cache of these fields is growing when we check it with :

/_stats/fielddata?fields=*&pretty

So for instance I add in the template this definition:
"itemUrl": {
"index": "not_analyzed",
"doc_values": true,
"type": "string"
}

and I see in the stats:
"events_54c06841-02e5-4e07-9908-da32b97be02d_2015-07-04" : {
"primaries" : {
"fielddata" : {
"memory_size_in_bytes" : 3455416,
"evictions" : 0,
"fields" : {
"itemUrl" : {
"memory_size_in_bytes" : 2589088
},
"itemId" : {
"memory_size_in_bytes" : 633912
},
"itemMimeMetaType" : {
"memory_size_in_bytes" : 1848
},
"clientId" : {
"memory_size_in_bytes" : 222032
},
"itemLabel" : {
"memory_size_in_bytes" : 8536
}
}
}
}

As you can see, "itemUrl" is present. Am I doing something wrong here ?

2

You could try adding the clause:

"fielddata": { "format": "doc_values"},

to your field mapping. This is another way to enable doc values

3

Hi Harlin,

Thanks for the quick answer. I have two questions:

  1. Is my method wrong in describing the "docs_value : true" ?
  2. How is your suggested configurations fits in the index mapping ? can you please provide example ?

Thanks !!

4

Hey Raz,

  1. No, your method is not wrong using "doc_values: true"

  2. Here's an example:

    "ts": { "fielddata": { "format": "doc_values"},"format": "dateOptionalTime","type": "date","doc_values": true}

Also, when you changed your mapping, did you re-index your data after applying the mapping? In order for doc values to take effect, every doc that fits the mapping has to be indexed after the mapping has been applied.

5

Hi Harlin.
Thank you for the example :smile:
No. I didn't re-index my data. I'm using daily indices. So from what I understood from the docs, fields that will be indexed after applying the mapping, will be stored as doc_values on the disk.
I don't mind that fields that were indexed before the change are not stored as doc_values on the disk (because in time everything will become doc_values because I will archive the old indices).
I also read that after segment merge, the changes will take effect on old indices.
Should I re-index anyway ?

6

So from what I understood from the docs, fields that will be indexed
after applying the mapping, will be stored as doc_values on the disk.

This is definitely true, since you are using daily indices, your new indices should be using doc values

I also read that after segment merge, the changes will take effect on old indices.

You may be right about this, but I wouldn't trust it.

I wouldn't re index if you don't have to, though it is the only way to grantee that all of your indices will use doc values.

7

Thank you for the comprehensive response !
Another small question. Can I define the doc_values in dynamic_template if I want that all the fields from certain type will use it ?
If for instance I define:

{
"my_type" : {
"dynamic_templates" : [
{
"template_1" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string",
"doc_values" :true,
"fielddata: {
"format" : "doc_values"
}
}
}
}
],
"properties": {
"my_field": {
"type": "string",
"index": "not_analyzed"
}
}
}
}

my_field will also be defined as doc_values ?