summaryrefslogtreecommitdiff
path: root/corpus/support/README
diff options
context:
space:
mode:
Diffstat (limited to 'corpus/support/README')
-rw-r--r--corpus/support/README2
1 files changed, 2 insertions, 0 deletions
diff --git a/corpus/support/README b/corpus/support/README
new file mode 100644
index 00000000..fdbd523e
--- /dev/null
+++ b/corpus/support/README
@@ -0,0 +1,2 @@
+Run ./tokenize.sh to tokenize text
+Edit eng_token_patterns and eng_token_list to add rules for things not to segment