Wildcard query limit?

Nick_Ricketts · May 3, 2013, 7:47pm

If I do a wildcard query containing only "*" on field "X" I only get back docs where field "X" is small. Field "X" is "not_analyzed".

Is there a size limit to field "X" when using wildcard query?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Nick_Ricketts · May 3, 2013, 11:32pm

Here is explain for the short one for field "X" that is finds.

ConstantScore(original:*
), product of:","details":[{"value":1.0,"description
":"boost"},{"value":0.8595584,"description":"queryNo
rm"}]}

Here is the explain for the field "X" that is long

{"value"
:0.0,"description":"no match on required clause (Con
stantScore(original:))","details":[{"value":0.0,"de
scription":"ConstantScore(original:) doesn't match
id 0"}]}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Nick_Ricketts · May 6, 2013, 4:46pm

Here is example of one that seems too long and wildcard wont rerun it even with "*"

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum iaculis lobortis aliquam. Maecenas tellus dui, sollicitudin tincidunt malesuada eget, placerat sed velit. Nunc non tempor nisl. Vestibulum lobortis eleifend est at feugiat. In in nulla eget mi tempus semper vitae non dolor. Nullam nisl nibh, consequat non euismod ultricies, egestas lobortis nibh. Vivamus lacinia pharetra ante id hendrerit. Ut vel nisl eget augue luctus vulputate sed elementum dolor. Nunc nec ante ante. Maecenas facilisis lobortis nisi scelerisque cursus. Fusce sed ante turpis. Praesent sit amet ante ac magna vehicula vehicula. Aenean aliquam, tortor a rhoncus tempor, eros massa pellentesque erat, vitae tincidunt nulla tellus vitae libero. Aliquam ac dui metus. Donec lobortis tortor eget dolor pharetra viverra. Suspendisse potenti. Sed elit quam, ultricies id laoreet vel, dignissim eget lorem. Maecenas tincidunt ante eros, non malesuada urna. Quisque id sodales turpis. Morbi et tellus quis leo auctor varius. Mauris commodo, metus vel vehicula tincidunt, massa arcu aliquam risus, a auctor purus lacus non diam. Proin pharetra placerat purus, quis venenatis sem laoreet ac. Donec enim magna, accumsan ac bibendum id, accumsan quis leo. In vitae sollicitudin nunc. Aliquam erat volutpat. Suspendisse mollis interdum ante in congue. Cras bibendum dictum ante dapibus porttitor. Pellentesque tempor, erat at aliquam luctus, lectus nunc commodo lacus, a dictum augue lorem vehicula erat. Proin in tellus quam, vel tincidunt arcu. Mauris sollicitudin ligula eget sapien consectetur euismod. Cras porta faucibus nisi, eu venenatis nibh molestie ut. Maecenas ultrices tempor est ac ultrices. Phasellus luctus rutrum orci non mollis. Sed tristique, arcu eu dictum auctor, nibh nibh luctus magna, lobortis cursus dolor diam ut enim. Suspendisse placerat mauris ac tellus porttitor ut mollis diam feugiat. Quisque sollicitudin ligula molestie sem ultricies ultrices. Praesent turpis sem, pharetra a sollicitudin sit amet, posuere eget purus. Donec a turpis at dolor vulputate adipiscing eu pharetra sem. Pellentesque tristique nulla sed magna tempus in convallis diam luctus. Aenean egestas tristique lectus eu faucibus. Praesent non nunc libero, at sollicitudin magna. Pellentesque mauris enim, hendrerit eu malesuada at, laoreet non felis. Duis vel nisi nisl, ut accumsan risus. Donec vel dignissim tortor. Nulla vestibulum tortor id enim blandit commodo. Aenean elementum, est eu gravida posuere, ligula lacus tristique erat, eu feugiat sem velit sed dolor. Sed id venenatis massa. Nulla vitae lorem vitae dolor iaculis vestibulum et non quam. Curabitur malesuada eros eu sem pulvinar sit amet congue mi hendrerit. Cras velit dui, venenatis a placerat at, mattis eu felis. Nulla orci tellus, euismod auctor vestibulum sed, laoreet ut diam. Maecenas mollis metus libero. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Mauris consequat velit ut erat consectetur aliquam. Nulla gravida porta arcu, auctor hendrerit ipsum fermentum nec. Nullam viverra lectus eu mi varius tincidunt. Maecenas volutpat urna a odio feugiat quis adipiscing nisl pretium. Aliquam erat volutpat. Pellentesque elementum tincidunt ligula, in auctor dolor ullamcorper sit amet. Cras ut mauris at lectus adipiscing consequat interdum sed magna. Duis bibendum lorem non nisi pharetra ultrices eget et lectus. Nunc venenatis, purus ut euismod tempor, nisi velit pretium leo, non molestie tellus justo eu neque. Praesent volutpat, massa quis sollicitudin vestibulum, risus nisi volutpat enim, sit amet facilisis nibh nisi sed lectus. Quisque accumsan, magna at euismod vestibulum, elit sem tincidunt libero, ut porta turpis quam sit amet est. Suspendisse odio magna, congue eu pellentesque at, gravida ac tellus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean vel orci in odio commodo hendrerit quis a sapien. Vivamus id bibendum diam. Morbi quis odio lacus, nec euismod dui. Cras auctor purus nec nulla convallis consectetur. Integer nec elit ut massa molestie vehicula vel et est. Vestibulum non lacus lorem, eu sollicitudin mauris. Donec nulla ligula, tincidunt in placerat eget, fermentum quis tellus. In libero ligula, interdum eget aliquam vitae, aliquam auctor tellus. Quisque auctor hendrerit eleifend. Morbi at nisi non tellus facilisis dapibus. Nam ac ligula erat. Morbi suscipit nibh sit amet erat scelerisque adipiscing. Curabitur accumsan convallis neque fermentum fermentum. Donec blandit, leo eu volutpat dignissim, arcu erat blandit quam, sit amet sodales felis erat in sem. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In hac habitasse platea dictumst.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jxerome · May 8, 2013, 3:19pm

Hello,

Lucene has a max_term_size which is 32766 bytes in Lucene 4.3.0 (and was
smaller in previous version.) Token longer are discarted and not indexed.
So your long values are not indexed and cannot be found.

More generaly, I'm not sure that using a wildcard like you do is a good
practice. If you want all documents, you should use a match_all query. If
you want document that have an original field, you should consider add a
boolean flag attribute to mark them. I don't propose you to use the
"missing" filter combined with the "not" filter because I guess they use
indexed terms (not tested).

Finally, since such big term will be discarded on indexation and there is
not good case in which they could be used in a query, I recommand you not
to index ("index" : "no" in mapping) them but just to keep them in your
source to get them back at search time.

Sincerly,
Jérôme Mainaud

Le samedi 4 mai 2013 01:32:20 UTC+2, Nick Ricketts a écrit :

Here is explain for the short one for field "X" that is finds.

ConstantScore(original:*
), product of:","details":[{"value":1.0,"description
":"boost"},{"value":0.8595584,"description":"queryNo
rm"}]}

Here is the explain for the field "X" that is long

{"value"
:0.0,"description":"no match on required clause (Con
stantScore(original:))","details":[{"value":0.0,"de
scription":"ConstantScore(original:) doesn't match
id 0"}]}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Nick_Ricketts · May 8, 2013, 3:28pm

Ya, I was curious of size limit. For this query we really use text/term since I do have text to match it against. The "*" query was just a simple query to post to forum to highlight the issue of field size limit being a possible issue for us down the road. Especially since it isn't mentioned anywhere I could find.

If we do run into that issue later is there a way to increase that limit?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jxerome · May 8, 2013, 7:50pm

This is a Lucene hard coded limit in IndexWriter.MAX_TERM_LENGTH which is
documented :

« Absolute hard maximum length for a term, in bytes once encoded as UTF8.
If a term arrives from the analyzer longer than this length, it is skipped
and a message is printed to infoStream, if set (see
IndexWriterConfig.setInfoStream(InfoStream)http://lucene.apache.org/core/4_3_0/core/org/apache/lucene/index/IndexWriterConfig.html#setInfoStream(org.apache.lucene.util.InfoStream)).
»

I don't know a way to change it.

If you need exact match on a long text, you maybe want to sha1 the text and
search on it. The term will be constant size and the search should be more
efficient.

Jérôme Mainaud

Le 8 mai 2013 17:28, "Nick Ricketts" nightshade427@gmail.com a écrit :

Ya, I was curious of size limit. For this query we really use text/term
since I do have text to match it against. The "*" query was just a simple
query to post to forum to highlight the issue of field size limit being a
possible issue for us down the road. Especially since it isn't mentioned
anywhere I could find.

If we do run into that issue later is there a way to increase that limit?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Nick_Ricketts · May 8, 2013, 8:00pm

I see now that it is documented in lucene now that I know what to look for. But, it isn't documented anywhere in elasticsearch that field should be less than 32766 bytes if you want to searchable in any way.

Thanks for the information, been a huge help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · May 8, 2013, 9:12pm

The term length limit is 32k characters.

There is no explicit field length limit exposed by Lucene 4 or ES any
more. You can control the maximum number of tokens per field with
LimitTokenCountAnalyzer.

http://lucene.apache.org/core/4_3_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/LimitTokenCountAnalyzer.html

Jörg

Am 08.05.13 22:00, schrieb Nick Ricketts:

I see now that it is documented in lucene now that I know what to look for. But, it isn't documented anywhere in elasticsearch that field should be less than 32766 bytes if you want to searchable in any way.

Thanks for the information, been a huge help.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.