diff options
| author | mjdenkowski <michael.j.denkowski@gmail.com> | 2014-03-11 15:47:04 -0400 |
|---|---|---|
| committer | mjdenkowski <michael.j.denkowski@gmail.com> | 2014-03-11 15:47:04 -0400 |
| commit | 48b44d3eeeb997abccda12149af584b196698316 (patch) | |
| tree | 43b654e42749ba767d882ab77dbeef96107e1beb /corpus | |
| parent | ba0a7d0cd688ee136c7c7b7776e68cb9603585b2 (diff) | |
| parent | de8ffd4598d6c1e45273b50642870a661b4bcad4 (diff) | |
Merge branch 'master' of github.com:redpony/cdec
Diffstat (limited to 'corpus')
| -rwxr-xr-x | corpus/support/quote-norm.pl | 1 | ||||
| -rw-r--r-- | corpus/support/token_list | 6 |
2 files changed, 5 insertions, 2 deletions
diff --git a/corpus/support/quote-norm.pl b/corpus/support/quote-norm.pl index 33604027..0366fad5 100755 --- a/corpus/support/quote-norm.pl +++ b/corpus/support/quote-norm.pl @@ -39,6 +39,7 @@ while(<STDIN>) { s/&\#([0-9]+);/pack("U", $1)/ge; # Regularlize spaces: + s/\x{ad}//g; # soft hyphen s/\x{a0}/ /g; # non-breaking space s/\x{2009}/ /g; # thin space s/\x{2028}/ /g; # "line separator" diff --git a/corpus/support/token_list b/corpus/support/token_list index 228663f6..d38638cf 100644 --- a/corpus/support/token_list +++ b/corpus/support/token_list @@ -156,8 +156,9 @@ Mass. Md. Mfg. Mgr. -Mexican-U.S. -Mich. +Mio. +Mrd. +Bio. Minn. Mo. Mon. @@ -187,6 +188,7 @@ Rd. Rev. R.J. C.L +Rs. Rte. Sat. W.T |