Pulling values for properties that have index set to "no"

Elastic Stack Elasticsearch
1

Hi All,

Been reading this (https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store) blog post on the Elastic.co site and I'm puzzled how you'd actually retrieve values stored as

"index": "no"

When _source and _all is disabled.

For instance, Kibana discover can't see the fields. The settings recognise the mappings are there.

Similarly the _search API (as per documentation) doesn't work without indexed fields and as far as I can tell, even less so if _all and _source is disabled.

So how would you retrieve these values?

2

Hi,

good question, as you said you can't retrieve the fields with the document Get API or by searching. But refering to the blog post you mentioned, the fields that are later used in aggregations are declared as having "doc_values": true in the mappings. This means that for those fields the data is stored on disk using the doc_values data structure which is optimized for use in aggregations but cannot be searched.

You can still take a look at those values using fielddata_fields functionality. Here is a small example:

PUT /test
{
  "mappings": {
    "type": {
      "_all":            { "enabled": false },
      "_source":         { "enabled": false },
      "properties": {
        "count": {
           "type": "integer", "doc_values": true, "index": "no" }
        
      }
    }
  }
}

PUT /test/type/1
{
  "count" : 5,
  "age": 21
}

PUT /test/type/2
{
  "count" : 10,
  "age" : 13
}

GET /test/type/1

GET /test/type/_search
{
  "query": {
    "ids": {
      "values" : ["1"]
    }
  },
  "fielddata_fields": ["count"]
}

With the above mapping, searching for "count" e.g. with a range query will give you a exception because the field is not indexed. However, using the "ids" query to filter one (or more) documents, you can see the field data (or in this case doc_values) of that field.

1 Like
3

@cbuescher - Thanks for the reply Christoph.

Ok, so I understand what you're saying and from 2.x doc_values were default enabled - So I'm confident the data is stored.

My original post was a little bit of a loaded question leading onto the fact I'm now trying to visualize the data that has been input.

I can see in the originally linked blog post screen captures demoing visualization but for the life of me can't figure out how he achieve that, there's a large chunk of information/guidance missing from what I can tell.

Looks like I wasn't the only one... http://engineering.laterooms.com/elasticsearch-as-a-time-series-database-part-2-getting-stats-in/

I suppose it leads on to two questions here:

  1. This maybe for the Kibana forum more, but does Kibana 4.5.x now support aggregations of doc_value fields?
  2. Is it unreasonable for me to have an expectation that once the data is loaded into the index (and it's correctly mapped) that I should just be able to use Kibana or (preferred) grafana to aggregate and visualize?

Many thanks,

jdmac

4

I cannot really answer that, those are indeed questions better asked to the kibana folks.

5

@cbuescher - Thanks again for the reply.

Unfortunately that's one annoyance of these discussion boards, having to repost topics when it crosses one member of the stack to another.

6

Just move the thread then, you can do that by editing the topic and picking a new category.

7

Fair point- hadn't realised you could. But still a restricting forum system
in terms of visibility when products for discussion work so tightly in a
stack. A better system would be topic tagging like serverfault and f5. Just
putting my suggestion out there.

8

It's a good point!