diff options
| author | Chris Dyer <cdyer@allegro.clab.cs.cmu.edu> | 2014-01-20 00:11:27 -0500 |
|---|---|---|
| committer | Chris Dyer <cdyer@allegro.clab.cs.cmu.edu> | 2014-01-20 00:11:27 -0500 |
| commit | 1915f029aa02f3528dafcd7b80a62a9c890b462b (patch) | |
| tree | ace3650fa8b1c30811ac21919b7be31af2567019 /corpus/support/fix-eos.pl | |
| parent | 8377889316984b27eebb7c03e812478f878dc84c (diff) | |
hindi edits
Diffstat (limited to 'corpus/support/fix-eos.pl')
| -rwxr-xr-x | corpus/support/fix-eos.pl | 10 |
1 files changed, 10 insertions, 0 deletions
diff --git a/corpus/support/fix-eos.pl b/corpus/support/fix-eos.pl new file mode 100755 index 00000000..584f8b46 --- /dev/null +++ b/corpus/support/fix-eos.pl @@ -0,0 +1,10 @@ +#!/usr/bin/perl -w +use strict; +use utf8; + +binmode(STDIN, ":utf8"); +binmode(STDOUT, ":utf8"); +while(<STDIN>) { + s/(\p{Devanagari}{2}[A-Za-z0-9! ,.\@\p{Devanagari}]+?)\s+(\.)(\s*$|\s+\|\|\|)/$1 \x{0964}$3/s; + print; +} |