Tokenizing terms with punctuation


(nurikabe) #1

Hi All,

I think I'm overlooking something in the docs: Wondering if there is a way
to define "special" tokens that include punctuation. For example, "C++",
"Yahoo!", etc.

Thanks!


(system) #2