# How expensive is the Source Filtering?

**URL:** <https://discuss.elastic.co/t/how-expensive-is-the-source-filtering/156164>\
**Category:** Elasticsearch\
**Created:** [November 10, 2018, 9:00pm UTC](https://discuss.elastic.co/t/how-expensive-is-the-source-filtering/156164 "2018-11-10T21:00:34Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![hym](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hym/32/22399_2.png) [@hym](https://discuss.elastic.co/u/hym)\
**Post date:** [November 10, 2018, 9:00pm UTC](https://discuss.elastic.co/t/how-expensive-is-the-source-filtering/156164/1 "2018-11-10T21:00:34Z")

</div>

Hi,  
Pretty big documents are stored in the Index but with not so much of fields indexable. let's say only 5% of fields are indexed but the rest just stored in \_source field.  
As mentioned the document is so big and when no Source Filtering is set, it takes a lot of time to have the result (just because of IO, not search). To have records in a reasonable time we decided to use Source filter but I am not sure how expensive is it for Elasticsearch (lucenc) to apply the projection. Please let me know if it would not be a source of performance issues or we should break down the document into more than one index.

Thanks in advance

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [November 11, 2018, 10:48am UTC](https://discuss.elastic.co/t/how-expensive-is-the-source-filtering/156164/2 "2018-11-11T10:48:10Z")

</div>

“Source” is stored as a blob of JSON in the underlying Lucene storage. It can be filtered but not without incurring the cost of reading the full JSON from Lucene.  
At index time you can choose to extract selected fields for storage in Lucene where they can be retrieved at query time individuallly without needing to parse full JSON. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-store.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-store.html)

---

<div class="post-metadata">

**Author:** ![hym](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hym/32/22399_2.png) [@hym](https://discuss.elastic.co/u/hym)\
**Post date:** [November 11, 2018, 11:03pm UTC](https://discuss.elastic.co/t/how-expensive-is-the-source-filtering/156164/3 "2018-11-11T23:03:35Z")

</div>

@Mark_Harwood thanks for replay.  
I just wanted to get some insights into the size of documents. We have document sizes of around 400KB (~1M 3 of them). I know it depends on underlying hardware, IO and etc. But is these kind of numbers are normal or our documents are too big?

One more thing, I take a look at the **Explain** and **Profile** APIs which I think are good more to investigate the index/search issues (if I got them right). Is there any other way to assess the IO times and numbers?

Thanks in advance

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 12, 2018, 12:42am UTC](https://discuss.elastic.co/t/how-expensive-is-the-source-filtering/156164/4 "2018-11-12T00:42:54Z")

</div>

Those are relatively large documents, at least based on my experience.

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [November 12, 2018, 8:41am UTC](https://discuss.elastic.co/t/how-expensive-is-the-source-filtering/156164/5 "2018-11-12T08:41:34Z")

</div>

> [@hym](#):
>
> Is there any other way to assess the IO times and numbers?

[Rally](https://github.com/elastic/rally) is our benchmarking tool

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 10, 2018, 8:41am UTC](https://discuss.elastic.co/t/how-expensive-is-the-source-filtering/156164/6 "2018-12-10T08:41:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.