diff options
| author | Patrick Simianer <p@simianer.de> | 2013-12-05 07:56:38 +0100 |
|---|---|---|
| committer | Patrick Simianer <p@simianer.de> | 2013-12-05 07:56:38 +0100 |
| commit | db6a6ecfa350cae29739c59df1210d8f76a479c9 (patch) | |
| tree | f137a001f57f170455c28ce97b5abb2726006cf6 /preprocess | |
init
Diffstat (limited to 'preprocess')
| -rwxr-xr-x | preprocess | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/preprocess b/preprocess new file mode 100755 index 0000000..716255d --- /dev/null +++ b/preprocess @@ -0,0 +1,5 @@ +#!/bin/zsh + +LANG=$1 +~/scripts/htmlentities 2>htmlentities.$LANG.err | ~/scripts/normalize-punctuation 2>normalize-punctuation.$LANG.err | ~/moses/scripts/tokenizer/tokenizer.perl -a -b -threads 1 -l $LANG 2>tokenizer.$LANG.err | ~/moses/scripts/tokenizer/lowercase.perl 2>lowercase.$LANG.err + |