# Using serialized doc\_value instead of \_source to improve read latency **URL:** \ **Category:** Elasticsearch\ **Created:** [April 20, 2015, 10:18pm UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334 "2015-04-20T22:18:03Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Itai\_Frenkel](https://avatars.discourse-cdn.com/v4/letter/i/ecc23a/32.png) [@Itai\_Frenkel](https://discuss.elastic.co/u/Itai_Frenkel)\ **Post date:** [April 20, 2015, 10:18pm UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/1 "2015-04-20T22:18:03Z")
Hi, We are having a performance problem in which for each hit, elasticsearch parses the entire \_source then generates a new Json with only the requested query \_source fields. In order to overcome this issue we would like to use mapping transform script that serializes the requested query fields (which is known in advance) into a doc\_value. Does that makes sense? The actual problem with the transform script is SecurityException that does not allow using any json serialization mechanism. A binary serialization would also be ok. Itai -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [April 20, 2015, 10:36pm UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/2 "2015-04-20T22:36:09Z")
This is how \_source works. doc\_values don't make sense in this regard - what you are looking for is using stored fields and have the transform script write to that. Loading stored fields (even one field per hit) may be slower than loading and parsing \_source, though. I'd just put this logic in the indexer, though. It will definitely help with other things as well, such as nasty huge mappings. Alternatively, find a way to avoid IO completely. How about using ES for search and something like riak for loading the actual data, if IO costs are so noticable? -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant [Lucene.NET](http://Lucene.NET) committer and PMC member On Mon, Apr 20, 2015 at 11:18 PM, Itai Frenkel [itaifrenkel@live.com](mailto:itaifrenkel@live.com) wrote: > Hi, > > We are having a performance problem in which for each hit, elasticsearch > parses the entire \_source then generates a new Json with only the requested > query \_source fields. In order to overcome this issue we would like to use > mapping transform script that serializes the requested query fields (which > is known in advance) into a doc\_value. Does that makes sense? > > The actual problem with the transform script is SecurityException that > does not allow using any json serialization mechanism. A binary > serialization would also be ok. > > Itai > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zsmri8LvzAqnXrwCA7B2PesCtH05BQxmj%3D3vMr%2B9abikw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zsmri8LvzAqnXrwCA7B2PesCtH05BQxmj%3D3vMr%2B9abikw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itai\_Frenkel](https://avatars.discourse-cdn.com/v4/letter/i/ecc23a/32.png) [@Itai\_Frenkel](https://discuss.elastic.co/u/Itai_Frenkel)\ **Post date:** [April 20, 2015, 11:14pm UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/3 "2015-04-20T23:14:03Z")
Itamar, 1. The \_source field includes many fields that are only being indexed, and many fields that are only needed as a query search result. \_source includes them both.The projection from \_source from the query result is too CPU intensive to do during search time for each result, especially if the size is big. 2. I agree that adding another NoSQL could solve this problem, however it is currently out of scope, as it would require syncing data with another data store. 3. Wouldn't a big stored field will bloat the lucene index size? Even if not, isn't non\_analyzed fields are destined to be (or already are) doc\_fields? On Tuesday, April 21, 2015 at 1:36:20 AM UTC+3, Itamar Syn-Hershko wrote: > This is how \_source works. doc\_values don't make sense in this regard - > what you are looking for is using stored fields and have the transform > script write to that. Loading stored fields (even one field per hit) may be > slower than loading and parsing \_source, though. > > I'd just put this logic in the indexer, though. It will definitely help > with other things as well, such as nasty huge mappings. > > Alternatively, find a way to avoid IO completely. How about using ES for > search and something like riak for loading the actual data, if IO costs are > so noticable? > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > [Lucene.NET](http://Lucene.NET) committer and PMC member > > On Mon, Apr 20, 2015 at 11:18 PM, Itai Frenkel \<[itaif...@live.com](mailto:itaif...@live.com) > \\> wrote: > > > Hi, > > > > We are having a performance problem in which for each hit, elasticsearch > > parses the entire \_source then generates a new Json with only the requested > > query \_source fields. In order to overcome this issue we would like to use > > mapping transform script that serializes the requested query fields (which > > is known in advance) into a doc\_value. Does that makes sense? > > > > The actual problem with the transform script is SecurityException that > > does not allow using any json serialization mechanism. A binary > > serialization would also be ok. > > > > Itai > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itai\_Frenkel](https://avatars.discourse-cdn.com/v4/letter/i/ecc23a/32.png) [@Itai\_Frenkel](https://discuss.elastic.co/u/Itai_Frenkel)\ **Post date:** [April 20, 2015, 11:18pm UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/4 "2015-04-20T23:18:09Z")
Also - does "fielddata": { "loading": "eager" } makes sense with doc\_values in this use case? Would that combination be supported in the future? On Tuesday, April 21, 2015 at 2:14:03 AM UTC+3, Itai Frenkel wrote: > Itamar, > > 1. The \_source field includes many fields that are only being indexed, and > many fields that are only needed as a query search result. \_source includes > them both.The projection from \_source from the query result is too CPU > intensive to do during search time for each result, especially if the size > is big. > 2. I agree that adding another NoSQL could solve this problem, however it > is currently out of scope, as it would require syncing data with another > data store. > 3. Wouldn't a big stored field will bloat the lucene index size? Even if > not, isn't non\_analyzed fields are destined to be (or already are) > doc\_fields? > > On Tuesday, April 21, 2015 at 1:36:20 AM UTC+3, Itamar Syn-Hershko wrote: > > > This is how \_source works. doc\_values don't make sense in this regard - > > what you are looking for is using stored fields and have the transform > > script write to that. Loading stored fields (even one field per hit) may be > > slower than loading and parsing \_source, though. > > > > I'd just put this logic in the indexer, though. It will definitely help > > with other things as well, such as nasty huge mappings. > > > > Alternatively, find a way to avoid IO completely. How about using ES for > > search and something like riak for loading the actual data, if IO costs are > > so noticable? > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > On Mon, Apr 20, 2015 at 11:18 PM, Itai Frenkel [itaif...@live.com](mailto:itaif...@live.com) wrote: > > > > > Hi, > > > > > > We are having a performance problem in which for each hit, elasticsearch > > > parses the entire \_source then generates a new Json with only the requested > > > query \_source fields. In order to overcome this issue we would like to use > > > mapping transform script that serializes the requested query fields (which > > > is known in advance) into a doc\_value. Does that makes sense? > > > > > > The actual problem with the transform script is SecurityException that > > > does not allow using any json serialization mechanism. A binary > > > serialization would also be ok. > > > > > > Itai > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d5abaeac-ff16-45ac-bb3d-62b53e497795%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d5abaeac-ff16-45ac-bb3d-62b53e497795%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [April 20, 2015, 11:24pm UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/5 "2015-04-20T23:24:58Z")
What if all those fields are collapsed to one, like you suggest, but that one field is projected out of \_source (think non-indexed json in a string field)? do you see a noticable performance gain then? What if that field is set to be stored (and loaded using fields, not via \_source)? what is the performance gain then? Fielddata and the doc\_values optimization on top of them will not help you here, those data structures aren't being used for sending data out, only for aggregations and sorting. Also, using fielddata will require indexing those fields; it is apparent that you are not looking to be doing that. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant [Lucene.NET](http://Lucene.NET) committer and PMC member On Tue, Apr 21, 2015 at 12:14 AM, Itai Frenkel [itaifrenkel@live.com](mailto:itaifrenkel@live.com) wrote: > Itamar, > > 1. The \_source field includes many fields that are only being indexed, and > many fields that are only needed as a query search result. \_source includes > them both.The projection from \_source from the query result is too CPU > intensive to do during search time for each result, especially if the size > is big. > 2. I agree that adding another NoSQL could solve this problem, however it > is currently out of scope, as it would require syncing data with another > data store. > 3. Wouldn't a big stored field will bloat the lucene index size? Even if > not, isn't non\_analyzed fields are destined to be (or already are) > doc\_fields? > > On Tuesday, April 21, 2015 at 1:36:20 AM UTC+3, Itamar Syn-Hershko wrote: > > > This is how \_source works. doc\_values don't make sense in this regard - > > what you are looking for is using stored fields and have the transform > > script write to that. Loading stored fields (even one field per hit) may be > > slower than loading and parsing \_source, though. > > > > I'd just put this logic in the indexer, though. It will definitely help > > with other things as well, such as nasty huge mappings. > > > > Alternatively, find a way to avoid IO completely. How about using ES for > > search and something like riak for loading the actual data, if IO costs are > > so noticable? > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > On Mon, Apr 20, 2015 at 11:18 PM, Itai Frenkel [itaif...@live.com](mailto:itaif...@live.com) wrote: > > > > > Hi, > > > > > > We are having a performance problem in which for each hit, elasticsearch > > > parses the entire \_source then generates a new Json with only the requested > > > query \_source fields. In order to overcome this issue we would like to use > > > mapping transform script that serializes the requested query fields (which > > > is known in advance) into a doc\_value. Does that makes sense? > > > > > > The actual problem with the transform script is SecurityException that > > > does not allow using any json serialization mechanism. A binary > > > serialization would also be ok. > > > > > > Itai > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuxvUoZ4L%2BUq0G82GLZKYfN-hj\_e\_gez6RsUc3hZeHbyw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuxvUoZ4L%2BUq0G82GLZKYfN-hj_e_gez6RsUc3hZeHbyw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itai\_Frenkel](https://avatars.discourse-cdn.com/v4/letter/i/ecc23a/32.png) [@Itai\_Frenkel](https://discuss.elastic.co/u/Itai_Frenkel)\ **Post date:** [April 20, 2015, 11:57pm UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/6 "2015-04-20T23:57:57Z")
A quick check shows there is no significant performance gain between doc\_value and stored field that is not a doc value. I suppose there are warm-up and file system caching issues are at play. I do not have that field in the source since the ETL process at this point does not generate it. The ETL could be fixed and then it will generate the required field. However, even then I would still prefer doc\_field over \_source since I do not need \_source at all. You are right to assume that reading the entire source parsing it and returning only one field would be fast (since the cpu is in the json generator I suspect, and not the parser, but that requires more work). On Tuesday, April 21, 2015 at 2:25:22 AM UTC+3, Itamar Syn-Hershko wrote: > What if all those fields are collapsed to one, like you suggest, but that > one field is projected out of \_source (think non-indexed json in a string > field)? do you see a noticable performance gain then? > > What if that field is set to be stored (and loaded using fields, not via > \_source)? what is the performance gain then? > > Fielddata and the doc\_values optimization on top of them will not help you > here, those data structures aren't being used for sending data out, only > for aggregations and sorting. Also, using fielddata will require indexing > those fields; it is apparent that you are not looking to be doing that. > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > [Lucene.NET](http://Lucene.NET) committer and PMC member > > On Tue, Apr 21, 2015 at 12:14 AM, Itai Frenkel \<[itaif...@live.com](mailto:itaif...@live.com) > \\> wrote: > > > Itamar, > > > > 1. The \_source field includes many fields that are only being indexed, > > and many fields that are only needed as a query search result. \_source > > includes them both.The projection from \_source from the query result is too > > CPU intensive to do during search time for each result, especially if the > > size is big. > > 2. I agree that adding another NoSQL could solve this problem, however it > > is currently out of scope, as it would require syncing data with another > > data store. > > 3. Wouldn't a big stored field will bloat the lucene index size? Even if > > not, isn't non\_analyzed fields are destined to be (or already are) > > doc\_fields? > > > > On Tuesday, April 21, 2015 at 1:36:20 AM UTC+3, Itamar Syn-Hershko wrote: > > > > > This is how \_source works. doc\_values don't make sense in this regard - > > > what you are looking for is using stored fields and have the transform > > > script write to that. Loading stored fields (even one field per hit) may be > > > slower than loading and parsing \_source, though. > > > > > > I'd just put this logic in the indexer, though. It will definitely help > > > with other things as well, such as nasty huge mappings. > > > > > > Alternatively, find a way to avoid IO completely. How about using ES for > > > search and something like riak for loading the actual data, if IO costs are > > > so noticable? > > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > On Mon, Apr 20, 2015 at 11:18 PM, Itai Frenkel [itaif...@live.com](mailto:itaif...@live.com) > > > wrote: > > > > > > > Hi, > > > > > > > > We are having a performance problem in which for each hit, > > > > elasticsearch parses the entire \_source then generates a new Json with only > > > > the requested query \_source fields. In order to overcome this issue we > > > > would like to use mapping transform script that serializes the requested > > > > query fields (which is known in advance) into a doc\_value. Does that makes > > > > sense? > > > > > > > > The actual problem with the transform script is SecurityException that > > > > does not allow using any json serialization mechanism. A binary > > > > serialization would also be ok. > > > > > > > > Itai > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [April 21, 2015, 12:05am UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/7 "2015-04-21T00:05:25Z")
Have you profiled it and seen that reading the source is actually the slow part? hot\_threads can lie here so I'd go with a profiler or just sigquit or something. I've got some reasonably big documents and generally don't see that as a problem even under decent load. I could see an argument for a second source field with the long stuff removed if you see the json decode or the disk read of the source be really slow - but transform doesn't do that. Nik On Mon, Apr 20, 2015 at 7:57 PM, Itai Frenkel [itaifrenkel@live.com](mailto:itaifrenkel@live.com) wrote: > A quick check shows there is no significant performance gain between > doc\_value and stored field that is not a doc value. I suppose there are > warm-up and file system caching issues are at play. I do not have that > field in the source since the ETL process at this point does not generate > it. The ETL could be fixed and then it will generate the required field. > However, even then I would still prefer doc\_field over \_source since I do > not need \_source at all. You are right to assume that reading the entire > source parsing it and returning only one field would be fast (since the cpu > is in the json generator I suspect, and not the parser, but that requires > more work). > > On Tuesday, April 21, 2015 at 2:25:22 AM UTC+3, Itamar Syn-Hershko wrote: > > > What if all those fields are collapsed to one, like you suggest, but that > > one field is projected out of \_source (think non-indexed json in a string > > field)? do you see a noticable performance gain then? > > > > What if that field is set to be stored (and loaded using fields, not via > > \_source)? what is the performance gain then? > > > > Fielddata and the doc\_values optimization on top of them will not help > > you here, those data structures aren't being used for sending data out, > > only for aggregations and sorting. Also, using fielddata will require > > indexing those fields; it is apparent that you are not looking to be doing > > that. > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > On Tue, Apr 21, 2015 at 12:14 AM, Itai Frenkel [itaif...@live.com](mailto:itaif...@live.com) wrote: > > > > > Itamar, > > > > > > 1. The \_source field includes many fields that are only being indexed, > > > and many fields that are only needed as a query search result. \_source > > > includes them both.The projection from \_source from the query result is too > > > CPU intensive to do during search time for each result, especially if the > > > size is big. > > > 2. I agree that adding another NoSQL could solve this problem, however > > > it is currently out of scope, as it would require syncing data with another > > > data store. > > > 3. Wouldn't a big stored field will bloat the lucene index size? Even if > > > not, isn't non\_analyzed fields are destined to be (or already are) > > > doc\_fields? > > > > > > On Tuesday, April 21, 2015 at 1:36:20 AM UTC+3, Itamar Syn-Hershko wrote: > > > > > > > This is how \_source works. doc\_values don't make sense in this regard - > > > > what you are looking for is using stored fields and have the transform > > > > script write to that. Loading stored fields (even one field per hit) may be > > > > slower than loading and parsing \_source, though. > > > > > > > > I'd just put this logic in the indexer, though. It will definitely help > > > > with other things as well, such as nasty huge mappings. > > > > > > > > Alternatively, find a way to avoid IO completely. How about using ES > > > > for search and something like riak for loading the actual data, if IO costs > > > > are so noticable? > > > > > > > > -- > > > > > > > > Itamar Syn-Hershko > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > Freelance Developer & Consultant > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > On Mon, Apr 20, 2015 at 11:18 PM, Itai Frenkel [itaif...@live.com](mailto:itaif...@live.com) > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > We are having a performance problem in which for each hit, > > > > > elasticsearch parses the entire \_source then generates a new Json with only > > > > > the requested query \_source fields. In order to overcome this issue we > > > > > would like to use mapping transform script that serializes the requested > > > > > query fields (which is known in advance) into a doc\_value. Does that makes > > > > > sense? > > > > > > > > > > The actual problem with the transform script is SecurityException > > > > > that does not allow using any json serialization mechanism. A binary > > > > > serialization would also be ok. > > > > > > > > > > Itai > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1Fs7xD63h0RXS8WZC-QvrnDOmfy6CUFB0VOZeCvXUHxQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1Fs7xD63h0RXS8WZC-QvrnDOmfy6CUFB0VOZeCvXUHxQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itai\_Frenkel](https://avatars.discourse-cdn.com/v4/letter/i/ecc23a/32.png) [@Itai\_Frenkel](https://discuss.elastic.co/u/Itai_Frenkel)\ **Post date:** [April 21, 2015, 2:20am UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/8 "2015-04-21T02:20:40Z")
Hi Nik, when \_source : true the time it takes for the search to complete in elasticsearch is very short. when \_souce is a list of fields it is significantly slower. Itai On Tuesday, April 21, 2015 at 3:06:06 AM UTC+3, Nikolas Everett wrote: > Have you profiled it and seen that reading the source is actually the slow > part? hot\_threads can lie here so I'd go with a profiler or just sigquit or > something. > > I've got some reasonably big documents and generally don't see that as a > problem even under decent load. > > I could see an argument for a second source field with the long stuff > removed if you see the json decode or the disk read of the source be really > slow - but transform doesn't do that. > > Nik > > On Mon, Apr 20, 2015 at 7:57 PM, Itai Frenkel \<[itaif...@live.com](mailto:itaif...@live.com) > \\> wrote: > > > A quick check shows there is no significant performance gain between > > doc\_value and stored field that is not a doc value. I suppose there are > > warm-up and file system caching issues are at play. I do not have that > > field in the source since the ETL process at this point does not generate > > it. The ETL could be fixed and then it will generate the required field. > > However, even then I would still prefer doc\_field over \_source since I do > > not need \_source at all. You are right to assume that reading the entire > > source parsing it and returning only one field would be fast (since the cpu > > is in the json generator I suspect, and not the parser, but that requires > > more work). > > > > On Tuesday, April 21, 2015 at 2:25:22 AM UTC+3, Itamar Syn-Hershko wrote: > > > > > What if all those fields are collapsed to one, like you suggest, but > > > that one field is projected out of \_source (think non-indexed json in a > > > string field)? do you see a noticable performance gain then? > > > > > > What if that field is set to be stored (and loaded using fields, not via > > > \_source)? what is the performance gain then? > > > > > > Fielddata and the doc\_values optimization on top of them will not help > > > you here, those data structures aren't being used for sending data out, > > > only for aggregations and sorting. Also, using fielddata will require > > > indexing those fields; it is apparent that you are not looking to be doing > > > that. > > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > On Tue, Apr 21, 2015 at 12:14 AM, Itai Frenkel [itaif...@live.com](mailto:itaif...@live.com) > > > wrote: > > > > > > > Itamar, > > > > > > > > 1. The \_source field includes many fields that are only being indexed, > > > > and many fields that are only needed as a query search result. \_source > > > > includes them both.The projection from \_source from the query result is too > > > > CPU intensive to do during search time for each result, especially if the > > > > size is big. > > > > 2. I agree that adding another NoSQL could solve this problem, however > > > > it is currently out of scope, as it would require syncing data with another > > > > data store. > > > > 3. Wouldn't a big stored field will bloat the lucene index size? Even > > > > if not, isn't non\_analyzed fields are destined to be (or already are) > > > > doc\_fields? > > > > > > > > On Tuesday, April 21, 2015 at 1:36:20 AM UTC+3, Itamar Syn-Hershko > > > > wrote: > > > > > > > > > This is how \_source works. doc\_values don't make sense in this regard > > > > > > > > > > - what you are looking for is using stored fields and have the transform > > > > > script write to that. Loading stored fields (even one field per hit) may be > > > > > slower than loading and parsing \_source, though. > > > > > > > > > > I'd just put this logic in the indexer, though. It will definitely > > > > > help with other things as well, such as nasty huge mappings. > > > > > > > > > > Alternatively, find a way to avoid IO completely. How about using ES > > > > > for search and something like riak for loading the actual data, if IO costs > > > > > are so noticable? > > > > > > > > > > -- > > > > > > > > > > Itamar Syn-Hershko > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > Freelance Developer & Consultant > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > On Mon, Apr 20, 2015 at 11:18 PM, Itai Frenkel [itaif...@live.com](mailto:itaif...@live.com) > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > We are having a performance problem in which for each hit, > > > > > > elasticsearch parses the entire \_source then generates a new Json with only > > > > > > the requested query \_source fields. In order to overcome this issue we > > > > > > would like to use mapping transform script that serializes the requested > > > > > > query fields (which is known in advance) into a doc\_value. Does that makes > > > > > > sense? > > > > > > > > > > > > The actual problem with the transform script is SecurityException > > > > > > that does not allow using any json serialization mechanism. A binary > > > > > > serialization would also be ok. > > > > > > > > > > > > Itai > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b897aba2-c250-4474-a03f-1d2a993baef9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/630a2998-e2a9-44a3-9c93-e692be2c2338%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8fd7a5d2-77c7-4758-8c28-82f517131660%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0aff9959-4c66-4b82-8e09-082b743642e3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0aff9959-4c66-4b82-8e09-082b743642e3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itai\_Frenkel](https://avatars.discourse-cdn.com/v4/letter/i/ecc23a/32.png) [@Itai\_Frenkel](https://discuss.elastic.co/u/Itai_Frenkel)\ **Post date:** [April 21, 2015, 2:21am UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/9 "2015-04-21T02:21:57Z")
If I could focus the question better : How do I whitelist a specific class in the groovy script inside transform ? On Tuesday, April 21, 2015 at 1:18:03 AM UTC+3, Itai Frenkel wrote: > Hi, > > We are having a performance problem in which for each hit, elasticsearch > parses the entire \_source then generates a new Json with only the requested > query \_source fields. In order to overcome this issue we would like to use > mapping transform script that serializes the requested query fields (which > is known in advance) into a doc\_value. Does that makes sense? > > The actual problem with the transform script is SecurityException that > does not allow using any json serialization mechanism. A binary > serialization would also be ok. > > Itai -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e925c3b6-b102-413c-a320-62f1c0ffcf99%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e925c3b6-b102-413c-a320-62f1c0ffcf99%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itai\_Frenkel](https://avatars.discourse-cdn.com/v4/letter/i/ecc23a/32.png) [@Itai\_Frenkel](https://discuss.elastic.co/u/Itai_Frenkel)\ **Post date:** [April 21, 2015, 9:35am UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/10 "2015-04-21T09:35:42Z")
The answer is these changes in elasticsearch.yml: script.groovy.sandbox.class\_whitelist: com.fasterxml.jackson.databind.ObjectMapper script.groovy.sandbox.package\_whitelist: com.fasterxml.jackson.databind for some reason these classes are not shaded even though the pom.xml does shade them. On Tuesday, April 21, 2015 at 5:21:58 AM UTC+3, Itai Frenkel wrote: > If I could focus the question better : How do I whitelist a specific > class in the groovy script inside transform ? > > On Tuesday, April 21, 2015 at 1:18:03 AM UTC+3, Itai Frenkel wrote: > > > Hi, > > > > We are having a performance problem in which for each hit, elasticsearch > > parses the entire \_source then generates a new Json with only the requested > > query \_source fields. In order to overcome this issue we would like to use > > mapping transform script that serializes the requested query fields (which > > is known in advance) into a doc\_value. Does that makes sense? > > > > The actual problem with the transform script is SecurityException that > > does not allow using any json serialization mechanism. A binary > > serialization would also be ok. > > > > Itai -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b7787495-500b-4ed7-b0e6-4fad7fda1aa2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7787495-500b-4ed7-b0e6-4fad7fda1aa2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:18am UTC](https://discuss.elastic.co/t/using-serialized-doc-value-instead-of--source-to-improve-read-latency/23334/11 "2017-07-06T00:18:35Z")