summaryrefslogtreecommitdiff
path: root/corpus/support/token_patterns
AgeCommit message (Expand)Author
2014-12-29finnish case markingsChris Dyer
2014-12-29finnish abbrevsChris Dyer
2013-01-22russian abbrevsChris Dyer
2013-01-21tokenizer support for utf8 patternsChris Dyer
2012-12-05slight tokenization bug fixChris Dyer
2012-11-14major mert clean up, stuff for simple system demoChris Dyer