diff options
| author | Chris Dyer <redpony@gmail.com> | 2014-01-17 04:07:22 -0500 |
|---|---|---|
| committer | Chris Dyer <redpony@gmail.com> | 2014-01-17 04:07:22 -0500 |
| commit | 1bc1a92c0f72fe4266182f9cb467b75e670a1dac (patch) | |
| tree | 46b0b6a278c82252aecccca4e8a2b5ecbb5bf728 /corpus/support/quote-norm.pl | |
| parent | 8bce53d2a076ef76f3fea34b2bddb0a713ec4ea8 (diff) | |
| parent | c13f5b9f6b5e80cfe7719b25f90d8d73d5ef3098 (diff) | |
Merge branch 'master' of https://github.com/redpony/cdec
Diffstat (limited to 'corpus/support/quote-norm.pl')
| -rwxr-xr-x | corpus/support/quote-norm.pl | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/corpus/support/quote-norm.pl b/corpus/support/quote-norm.pl index 57f4ad77..7fe33db4 100755 --- a/corpus/support/quote-norm.pl +++ b/corpus/support/quote-norm.pl @@ -61,6 +61,11 @@ while(<STDIN>) { s/«/"/g; s/»/"/g; tr/！-～/!-~/; + tr/०-९/0-9/; # devangari + tr/౦-౯/0-9/; # telugu + tr/೦-೯/0-9/; # kannada + tr/೦-௯/0-9/; # tamil + tr/൦-൯/0-9/; # malayalam s/、/,/g; # s/。/./g; s/…/.../g; |