# Scan query that returns document values only is heavily accessing the \*.FDT file
**URL:** https://discuss.elastic.co/t/scan-query-that-returns-document-values-only-is-heavily-accessing-the-fdt-file/20907
**Category:** Elasticsearch
**Created:** [November 23, 2014, 10:52pm UTC](https://discuss.elastic.co/t/scan-query-that-returns-document-values-only-is-heavily-accessing-the-fdt-file/20907 "2014-11-23T22:52:49Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Tzahi](https://discuss.elastic.co/u/Tzahi)
#### Post date: [November 23, 2014, 10:52pm UTC](https://discuss.elastic.co/t/scan-query-that-returns-document-values-only-is-heavily-accessing-the-fdt-file/20907/1 "2014-11-23T22:52:49Z")
Hi all,
I have a tests index with 43 million documenst. there is a string document
value for each document. (about 5-10 character value for each document)
Mapping is:
{
"myindex" : {
```
"mappings" : {
"num_type" : {
"_type" : {
"store" : true
},
"properties" : {
"doc_value" : {
"type" : "string",
"doc_values_format" : "default"
},
"int1" : {
"type" : "integer",
"index" : "analyzed",
"store" : true
},
"int2" : {
```
.
.
.
I need to retrieve the document values only for queries that may return
about 100,000 documents result set. I do not need ranking or anything else
that will slow this down.
My understanding is that if the query is only a filter – ranking is not
computed, and it is faster.
Here is a small python program to test it:
\*import \*elasticsearch
es = elasticsearch.Elasticsearch()
results = es.search(_"myindex"_, _"num\_type"_,
{
_"fields"_:[_"doc\_value"_],
_"size"_:1000,
_"query"_: {_"filtered"_: {
_"query"_: {_"match\_all"_:{}}
,_"filter"_: {
_"term"_: {_"r\_int3"_: 929}}
}}
},scroll=_"10s"_,search\_type=_"scan"_)
_while True_:
results = es.scroll(results[_"\_scroll\_id"_], scroll=_"10s"_)
_if len(results["hits"_][_"hits"_]) \<= 0:
_break_
The query runs pretty slow, and I see there is huge number of access to the
\*.fdt (field data) file.
But I ask for a document value field – so why does ES access the \*.fdt.
Thanks a lot in advance.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/89480f13-b00e-4e3f-a538-15fdbd18f073%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/89480f13-b00e-4e3f-a538-15fdbd18f073%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [November 24, 2014, 8:10am UTC](https://discuss.elastic.co/t/scan-query-that-returns-document-values-only-is-heavily-accessing-the-fdt-file/20907/2 "2014-11-24T08:10:41Z")
Doc values are stored in the .fdt files.
Jörg
On Sun, Nov 23, 2014 at 11:52 PM, Tzahi jakubovitz [tzahij@hotmail.com](mailto:tzahij@hotmail.com)
wrote:
> Hi all,
>
> I have a tests index with 43 million documenst. there is a string document
> value for each document. (about 5-10 character value for each document)
>
> Mapping is:
>
> {
>
> "myindex" : {
>
> ```
> "mappings" : {
>
> "num_type" : {
>
> "_type" : {
>
> "store" : true
>
> },
>
> "properties" : {
>
> "doc_value" : {
>
> "type" : "string",
>
> "doc_values_format" : "default"
>
> },
>
> "int1" : {
>
> "type" : "integer",
>
> "index" : "analyzed",
>
> "store" : true
>
> },
>
> "int2" : {
>
> ```
>
> .
>
> .
>
> .
>
> I need to retrieve the document values only for queries that may return
> about 100,000 documents result set. I do not need ranking or anything else
> that will slow this down.
>
> My understanding is that if the query is only a filter – ranking is not
> computed, and it is faster.
>
> Here is a small python program to test it:
>
> \*import \*elasticsearch
>
> es = elasticsearch.Elasticsearch()
>
> results = es.search(_"myindex"_, _"num\_type"_,
> {
> _"fields"_:[_"doc\_value"_],
> _"size"_:1000,
> _"query"_: {_"filtered"_: {
> _"query"_: {_"match\_all"_:{}}
> ,_"filter"_: {
> _"term"_: {_"r\_int3"_: 929}}
> }}
> },scroll=_"10s"_,search\_type=_"scan"_)
>
> _while True_:
> results = es.scroll(results[_"\_scroll\_id"_], scroll=_"10s"_)
> _if len(results["hits"_][_"hits"_]) \<= 0:
> _break_
>
> The query runs pretty slow, and I see there is huge number of access to
> the \*.fdt (field data) file.
>
> But I ask for a document value field – so why does ES access the \*.fdt.
>
> Thanks a lot in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/89480f13-b00e-4e3f-a538-15fdbd18f073%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/89480f13-b00e-4e3f-a538-15fdbd18f073%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/89480f13-b00e-4e3f-a538-15fdbd18f073%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/89480f13-b00e-4e3f-a538-15fdbd18f073%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEsDnXCbmV0tGmNwuYvAwdW-t%2BYJhf6mYmbN4ZVf3fMrQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEsDnXCbmV0tGmNwuYvAwdW-t%2BYJhf6mYmbN4ZVf3fMrQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Tzahi](https://discuss.elastic.co/u/Tzahi)
#### Post date: [November 24, 2014, 9:04am UTC](https://discuss.elastic.co/t/scan-query-that-returns-document-values-only-is-heavily-accessing-the-fdt-file/20907/3 "2014-11-24T09:04:01Z")
Thanks
Sorry - I did not stress this is _document_ values and not _field_ values.
Document values are stores in DVD file. which is small, compressed format.
I defined it to avoide having to access and parse the lucene document from
the huge FDT file (in my test- FDT file is 1000 times bigger than DVD file).
see
[https://lucene.apache.org/core/4\_3\_1/core/org/apache/lucene/codecs/lucene42/Lucene42DocValuesFormat.html](https://lucene.apache.org/core/4_3_1/core/org/apache/lucene/codecs/lucene42/Lucene42DocValuesFormat.html)
.
I still try to avoide accessing the FDT file - it makes my query toooo slow.
Thanks again.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cd6ed6a9-f1c7-47c4-be3d-833553cb2bf6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cd6ed6a9-f1c7-47c4-be3d-833553cb2bf6%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [November 24, 2014, 9:13am UTC](https://discuss.elastic.co/t/scan-query-that-returns-document-values-only-is-heavily-accessing-the-fdt-file/20907/4 "2014-11-24T09:13:17Z")
Oh, sorry. Yess, doc values are in .dvd files.
I assume that ES still puts hidden "type" and "uid" field in .fdt. But I'm
also surprised, there should be not much disk access for that.
Jörg
On Mon, Nov 24, 2014 at 10:04 AM, Tzahi jakubovitz [tzahij@hotmail.com](mailto:tzahij@hotmail.com)
wrote:
> Thanks
> Sorry - I did not stress this is _document_ values and not _field_ values.
> Document values are stores in DVD file. which is small, compressed format.
> I defined it to avoide having to access and parse the lucene document from
> the huge FDT file (in my test- FDT file is 1000 times bigger than DVD file).
> see
> [Lucene42DocValuesFormat (Lucene 4.3.1 API)](https://lucene.apache.org/core/4_3_1/core/org/apache/lucene/codecs/lucene42/Lucene42DocValuesFormat.html)
> .
>
> I still try to avoide accessing the FDT file - it makes my query toooo
> slow.
>
> Thanks again.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/cd6ed6a9-f1c7-47c4-be3d-833553cb2bf6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cd6ed6a9-f1c7-47c4-be3d-833553cb2bf6%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/cd6ed6a9-f1c7-47c4-be3d-833553cb2bf6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/cd6ed6a9-f1c7-47c4-be3d-833553cb2bf6%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEnzt3BFr-6jmQ6voNxn9pkG5bsdYnK-iV8HauRTRkKyA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEnzt3BFr-6jmQ6voNxn9pkG5bsdYnK-iV8HauRTRkKyA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:48am UTC](https://discuss.elastic.co/t/scan-query-that-returns-document-values-only-is-heavily-accessing-the-fdt-file/20907/5 "2017-07-06T00:48:10Z")