Ignore_above for text datatype in elasticsearch

Elastic Stack Elasticsearch
1

In elasticsearch 5 string datatype has been removed as explained nicely in this blog post.

ignore_above is not supported with text datatype. Can someone let me know how does text field is then protected against Lucene’s term byte-length limit as explained in the documentation here, especially the below section

This option is also useful for protecting against Lucene’s term byte-length limit of 32766 .

2

Anyone any thoughts on this? Same has been asked in stackoverflow also and no response there also.

3

Maybe by using the token length filter in an analyzer config?

4

Thanks @Mark_Harwood for replying.

So this I have to specify in analyzer config or by default elasticsearch does it for text field ?

5

It's not on by default.
I've not generally had a need for it because most text gets sliced up by tokenizers into smaller tokens anyway based on whitespace, punctuation etc.
Maybe a base64 encoded image might produce a single big token but I've not ran into content like that

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.