Visualization Filter Missing Rows when Quering Long Strings

[ofernachum (Ofer)]

Hey folks,

Im running Kiban 5.2.1 with Elasticsearch 5.2.1.

I have an Index with mapping that include string field containing long strings. For example:

Path.keyword =
"{ORS}{FGG only}{/D444/ORS Modifications/EIXXS ORP REQ001122}{EIXXX-CXXX REQDFESK}{.....}{Displayignore_above is the character count, but Lucene counts bytes}{VBBGF KJHKH JKHKJH REQ17004}{Alert Display REQ17076}{page numbering REQ22334}{page number format shall be as folowing: D/D First D: currently shown CAS page out of all available pages range - according to currently avaliable number of pages Power up/factory default - 1 Second D: amount of all available pages REQ17}"

When executing Discovery searchs, Kiban returns all the matching records.

Hoever, in Visualization, if I create a filter to match this string with a prefix, only records which have the stored string's lenth below ~ 250 charecters included in the filtered visualization.

For example, let's assume I have an index with only 2 records:

ID: 1
Path.keyword: "{ORS}{FGG onl}{Very Long String}{Test A}{Text B}"

ID: 2
Path.keyword: "{ORS}{FGG onl}{Text C}"

If I just search in descovery for "{ORS}{FGG onl}" Kibana will show both records, as the pattern "{ORS}{FGG onl}" is common for both.

If I create a visualization, based on the above search (after saving it), and add to the visualization the following filter, for example (Which should return both records as they poth starts with"{ORS}{FGG onl}" )

{
"query": {
"prefix": { "Path.keyword": "{ORS}{FGG onl}"
},
"require_field_match": false
}

Kiban just ignor's record ID: 2 ... as it looks filters just ignore strings byond certain length.

Appreciate any assistance overcoming this issue.

Cheers, Ofer N.

[timroes]

Hi Ofer,

Elasticsearch has something called "ignore_above" in the mapping, that can be used to set a length, above which the strings aren't indexed anymore for search. This is by default set to something like 256, which would explain your behavior - except, that it is still found in Discover. (See also: https://www.timroes.de/2016/05/29/elasticsearch-kibana-queries-in-depth-tutorial/#elasticseach-doesn-t-find-terms-in-long-fields)

Prior to Elasticsearch 5.1 you had something called the _all field, which was used for searches without a fieldname. So the documents would have been found in Discover, if you would have omitted the field name. Since 5.1 this isn't the case anymore and I am not 100% sure, why your search in Discover did reveal the documents. Did you use the fieldname when searching for the value?

In any case: if you know you have a field containing long values, you need to search for, you should increase the ignore_above value of that field in the mapping, see: https://www.elastic.co/guide/en/elasticsearch/reference/5.2/ignore-above.html

Cheers,
Tim