summaryrefslogtreecommitdiff
path: root/corpus/support/tokenizer.pl
AgeCommit message (Expand)Author
2014-01-15deal with hindiChris Dyer
2013-09-05Unbuffered mode, flush after each line where possible, skip otherwiseMichael Denkowski
2013-04-19hindiChris Dyer
2013-01-21tokenizer support for utf8 patternsChris Dyer
2013-01-21a little bit of cleanupChris Dyer
2012-12-05remove logging, you should be using pvChris Dyer
2012-11-14major mert clean up, stuff for simple system demoChris Dyer