summaryrefslogtreecommitdiff
path: root/corpus/support/tokenizer.pl
AgeCommit message (Expand)Author
2015-05-21deal with curly quotesChris Dyer
2014-12-29deal with eur symbolChris Dyer
2014-09-15migrate to new Cython versionChris Dyer
2014-03-18chris editsChris Dyer
2014-01-20deal with acronyms in hindiChris Dyer
2014-01-20hindi editsChris Dyer
2014-01-15deal with hindiChris Dyer
2013-09-05Unbuffered mode, flush after each line where possible, skip otherwiseMichael Denkowski
2013-04-19hindiChris Dyer
2013-01-21tokenizer support for utf8 patternsChris Dyer
2013-01-21a little bit of cleanupChris Dyer
2012-12-05remove logging, you should be using pvChris Dyer
2012-11-14major mert clean up, stuff for simple system demoChris Dyer