First I want to say that I really like the api of this library, it is very easy to use.
What I want to propose:
Would be great if we could specify specific tokens to be matched case insensitive.
Lets say I have a token called WHERE, would be great if it also matched where.
i.e.
Tokenizer.DefineTokens(TOKEN_TYPE, []string{"WHERE"}, true)
or
Tokenizer.DefineInsensitiveTokens(TOKEN_TYPE, []string{"WHERE"})