summaryrefslogtreecommitdiff
path: root/corpus
diff options
context:
space:
mode:
authorChris Dyer <cdyer@cs.cmu.edu>2012-11-16 00:24:52 -0500
committerChris Dyer <cdyer@cs.cmu.edu>2012-11-16 00:24:52 -0500
commit1cd434c2c971f23bc9fcbe18062b24c02873126d (patch)
tree6f5a7f12d89200724174a2b562f702f6b5744432 /corpus
parent6e6a9017e3554c478bc9f53fa28b113b2eb6350a (diff)
fix
Diffstat (limited to 'corpus')
-rw-r--r--corpus/README.md5
1 files changed, 5 insertions, 0 deletions
diff --git a/corpus/README.md b/corpus/README.md
index 935d9a65..adc35b84 100644
--- a/corpus/README.md
+++ b/corpus/README.md
@@ -5,11 +5,13 @@ Many of these scripts assume that the input is [UTF-8 encoded](http://en.wikiped
## Paste parallel files together
This script reads one line at a time from a set of files and concatenates them with a triple pipe separator (`|||`) in the output. This is useful for generating parallel corpora files for training or evaluation:
+
./paste-files.pl file.a file.b file.c [...]
## Punctuation Normalization and Tokenization
This script tokenizes text in any language (well, it does a good job in most languages, and in some it will completely go crazy):
+
./tokenize-anything.sh < input.txt > output.txt
It also normalizes a lot of unicode symbols and even corrects some common encoding errors. It can be applied to monolingual and parallel corpora directly.
@@ -17,16 +19,19 @@ It also normalizes a lot of unicode symbols and even corrects some common encodi
## Text lowercasing
This script also does what it says, provided your input is in UTF8:
+
./lowercase.pl < input.txt > output.txt
## Length ratio filtering (for parallel corpora)
This script computes statistics about sentence length ratios in a parallel corpus and removes sentences that are statistical outliers. This tends to remove extremely poorly aligned sentence pairs or sentence pairs that would otherwise be difficult to align:
+
./filter-length.pl input.src-trg > output.src-trg
## Add infrequent self-transaltions to a parallel corpus
This script identifies rare words (those that occur less than 2 times in the corpus) and which have the same orthographic form in both the source and target language. Several copies of these words are then inserted at the end of the corpus that is written, which improves alignment quality.
+
./add-self-translations.pl input.src-trg > output.src-trg