diff options
| author | Chris Dyer <cdyer@allegro.clab.cs.cmu.edu> | 2013-03-08 22:44:49 -0500 |
|---|---|---|
| committer | Chris Dyer <cdyer@allegro.clab.cs.cmu.edu> | 2013-03-08 22:44:49 -0500 |
| commit | 3a162d28033d1b9d5241e31f32978dba4eba6296 (patch) | |
| tree | ada0ec536f5a1835cc1507784ca3c8798636d273 /corpus/support/quote-norm.pl | |
| parent | 9f0109076f1c95170cbe46e1708597bc6e0f9fd4 (diff) | |
few preproc fixes
Diffstat (limited to 'corpus/support/quote-norm.pl')
| -rwxr-xr-x | corpus/support/quote-norm.pl | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/corpus/support/quote-norm.pl b/corpus/support/quote-norm.pl index d2980092..b104e73c 100755 --- a/corpus/support/quote-norm.pl +++ b/corpus/support/quote-norm.pl @@ -11,6 +11,8 @@ while(<STDIN>) { s/&\s*squot\s*;/'/gi; s/&\s*quot\s*;/"/gi; s/&\s*amp\s*;/&/gi; + s/&\s*nbsp\s*;/&/gi; + s/&\s*#\s*160\s*;/ /gi; s/ (\d\d): (\d\d)/ $1:$2/g; s/[\x{20a0}]\x{20ac}]/ EUR /g; s/[\x{00A3}]/ GBP /g; |