Ignore_above for text datatype in elasticsearch

[Deb]

In elasticsearch 5 string datatype has been removed as explained nicely in this blog post.

ignore_above is not supported with text datatype. Can someone let me know how does text field is then protected against Lucene’s term byte-length limit as explained in the documentation here, especially the below section

This option is also useful for protecting against Lucene’s term byte-length limit of 32766 .

[Deb]

Anyone any thoughts on this? Same has been asked in stackoverflow also and no response there also.

[Mark_Harwood]

@Deb

how does text field is then protected against Lucene’s term byte-length limit

Maybe by using the token length filter in an analyzer config?

[Deb]

Thanks @Mark_Harwood for replying.

So this I have to specify in analyzer config or by default elasticsearch does it for text field ?

[Mark_Harwood]

It's not on by default.
I've not generally had a need for it because most text gets sliced up by tokenizers into smaller tokens anyway based on whitespace, punctuation etc.
Maybe a base64 encoded image might produce a single big token but I've not ran into content like that