# Hash matching queries **URL:** \ **Category:** Elasticsearch\ **Created:** [June 6, 2012, 3:02pm UTC](https://discuss.elastic.co/t/hash-matching-queries/7995 "2012-06-06T15:02:12Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Alex\_4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_4/32/2855_2.png) [@Alex\_4](https://discuss.elastic.co/u/Alex_4)\ **Post date:** [June 6, 2012, 3:02pm UTC](https://discuss.elastic.co/t/hash-matching-queries/7995/1 "2012-06-06T15:02:12Z")
Hello elasticsearch members, I am currently have in my elasticsearch documents that have a field with a string of the form "hash\_1 hash\_2 hash\_3 ... hash\_n". Each document has roughly around 5000 to 6000 hashes in this field. Currently, my local elasticsearch has about 20000 of these documents. I query elasticsearch with a string "queryhash\_1 queryhash\_2 ... queryhash\_m", which can be anywhere between 200 to 1000 hashes. I was wondering if there are any configurations that could help me better solve my problem. I am currently using query string queries, with whitespace tokenizers and analyzers. I am also open to ideas on restructuring/reindexing the documents. Thank you! ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 10, 2012, 9:26pm UTC](https://discuss.elastic.co/t/hash-matching-queries/7995/2 "2012-06-10T21:26:38Z")
First, though effectively not a big difference, is to use multiple values for the field, and have it not\_analyzed. Its nicer (i.e. "hash" : ["hash\_1", "hash\_2"]. When searching, you can use terms query with the list of terms (and thus no need to parse the query\_string). On Wed, Jun 6, 2012 at 5:02 PM, Alex [alexahnhome@gmail.com](mailto:alexahnhome@gmail.com) wrote: > Hello elasticsearch members, > > I am currently have in my elasticsearch documents that have a field with a > string of the form "hash\_1 hash\_2 hash\_3 ... hash\_n". Each document has > roughly around 5000 to 6000 hashes in this field. Currently, my local > elasticsearch has about 20000 of these documents. I query elasticsearch > with a string "queryhash\_1 queryhash\_2 ... queryhash\_m", which can be > anywhere between 200 to 1000 hashes. I was wondering if there are any > configurations that could help me better solve my problem. I am currently > using query string queries, with whitespace tokenizers and analyzers. I am > also open to ideas on restructuring/reindexing the documents. Thank you! ---
**Author:** ![Alex\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_3/32/2855_2.png) [@Alex\_3](https://discuss.elastic.co/u/Alex_3)\ **Post date:** [June 11, 2012, 1:51pm UTC](https://discuss.elastic.co/t/hash-matching-queries/7995/3 "2012-06-11T13:51:12Z")
> Thank you for responding! By "effectively not a big difference" do you > mean that it will not decrease query time all that much? My plan is to one > day have millions of these documents, and I was wondering if query times > would increase with a growth in number of documents. I am really hoping > that the query times scale with the query length, and not the number of > documents. I've read articles online of people achieving indexing rates of > around 1000 documents a second. I am only getting around 20-40 a second, is > this to be expected because of the long hash strings? I've tried setting > the refresh interval to -1 during indexing, but it did not make a > significant difference (perhaps it will when I have millions of documents). > I hope to hear from you again, thanks! ---
**Author:** ![Alex\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_3/32/2855_2.png) [@Alex\_3](https://discuss.elastic.co/u/Alex_3)\ **Post date:** [June 11, 2012, 7:19pm UTC](https://discuss.elastic.co/t/hash-matching-queries/7995/4 "2012-06-11T19:19:44Z")
For clarification, Documents are put into ES from JSON dump files. I expect to receive and inject these JSON files into ES consistently. New documents do not invalidate older ones, and each should be treated as a unique document. I am essentially trying to solve a clustering/categorization/prediction problem, where given some input, I want to find the most similar documents to it (think k nearest neighbors). Each hash for a document represents an attribute for the document, so the more hashes a query has in common with a document, the more attributes it has in common with the document. What I want from a query (which ES is doing right now, though a bit slowly), is to return all the documents above a certain match % (for example, 10% of the number of hashes in the query) - or for simplicity, just above some constant threshold number. The issues I am running into right now are that indexing is taking a very long time, and querying is also taking quite a while (anywhere from two to ten seconds). Any suggestions would be appreciated, thank you. ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 11, 2012, 10:40pm UTC](https://discuss.elastic.co/t/hash-matching-queries/7995/5 "2012-06-11T22:40:40Z")
Hi Alex We discussed this at length on IRC, but then you timed out. Try this query: [gist:2a127562eb0808408097 ยท GitHub](https://gist.github.com/2a127562eb0808408097) Your query needs are quite different from normal full text search, so you will need to tune your mapping accordingly. You might get some gain from indexing your 'hashes' as integers rather than as text. Also, you probably want to tweak things like disabling norms, disabling term frequency etc, and you may want to use span queries instead of normal term or text queries. I'd suggest buying a book like Lucene in Action to understand more about what is happening under the covers. I think that's the only way you're going to get really good results with the type of queries you want to run. clint On Mon, 2012-06-11 at 06:51 -0700, Alex wrote: > ``` > Thank you for responding! By "effectively not a big > difference" do you mean that it will not decrease query time > all that much? My plan is to one day have millions of these > documents, and I was wondering if query times would increase > with a growth in number of documents. I am really hoping that > the query times scale with the query length, and not the > number of documents. I've read articles online of people > achieving indexing rates of around 1000 documents a second. I > am only getting around 20-40 a second, is this to be expected > because of the long hash strings? I've tried setting the > refresh interval to -1 during indexing, but it did not make a > significant difference (perhaps it will when I have millions > of documents). I hope to hear from you again, thanks! > > ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:24am UTC](https://discuss.elastic.co/t/hash-matching-queries/7995/6 "2017-07-06T03:24:52Z")