README.md


1
2
3
4
5
6
7
8
9
10
11
12
13

scripts
=======

A number of NLP related scripts.
Some scripts require my zipf gem,
see https://github.com/pks/zipf

compound-splitter.perl and tokenizer.no-escape.perl
taken from the moses [1] toolkit.


[1] https://github.com/moses-smt/mosesdecoder