nlp_scripts/preprocess, branch master Mirror of https://github.com/pks/nlp_scripts.git https://git.simianer.de/mirrored/nlp_scripts/atom?h=master 2026-02-26T10:05:59Z overhaul 2026-02-26T10:05:59Z Patrick Simianer patrick@lilt.com 2026-02-26T10:05:59Z urn:sha1:b31ace79ea5f6b3f279c544cd3a443d6fbf2a24d mv 2016-07-05T09:01:46Z Patrick Simianer p@simianer.de 2016-07-05T09:01:46Z urn:sha1:2b1d7f881c19c4d4b5afae194e02d3300c7675d0 fix 2015-01-07T18:13:09Z Patrick Simianer p@simianer.de 2015-01-07T18:13:09Z urn:sha1:77ca2f06c1f1ee9689138e1edf7d66e0214c231b steal tokenizer from moses' scripts 2014-06-14T12:43:14Z Patrick Simianer p@simianer.de 2014-06-14T12:43:14Z urn:sha1:2783f837303ae07c4a1d676302bca779abbb1296 fix 2014-03-17T12:50:41Z Patrick Simianer p@simianer.de 2014-03-17T12:50:41Z urn:sha1:b08be6207a78fc3639231489b74b4c72d78221d7 a lot of ... and --- cause moses' compound splitter to hang 2014-03-17T12:13:09Z Patrick Simianer p@simianer.de 2014-03-17T12:13:09Z urn:sha1:3bbbd92fc7b1f9d0d61f8573fee0d0b2f40960d3 no non printables in preproc 2014-03-16T18:17:24Z Patrick Simianer p@simianer.de 2014-03-16T18:17:24Z urn:sha1:224be546596aae5fe3bcd9c46d3eac3f84b386fd make use of nlp_ruby, LICENSE 2014-01-29T18:14:08Z Patrick Simianer p@simianer.de 2014-01-29T18:14:08Z urn:sha1:68acbb9a0c7967cb90a7e3756fc94fdd8a73d154 fix 2013-12-11T16:04:56Z Patrick Simianer simianer@cl.uni-heidelberg.de 2013-12-11T16:04:56Z urn:sha1:85952929c7f4451a65597b589deade7b6c63403a use moses tokenizer but without escaping of 'special' chars 2013-12-11T15:56:56Z Patrick Simianer simianer@cl.uni-heidelberg.de 2013-12-11T15:56:56Z urn:sha1:941da973ffd77b5b0accb64cb4ffe7f593dcd5f8