1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
Language model inference code by Kenneth Heafield <infer at kheafield.com>
See LICENSE for list of files by other people and their licenses.
Compile: ./compile.sh
Run: ./query lm/test.arpa <text
Build binary format: ./build_binary lm/test.arpa test.binary
Use binary format: ./query test.binary <text
Test (uses Boost): ./test.sh
Currently, it loads an ARPA file in 2/3 the time SRI takes and uses 6.5 GB when SRI takes 11 GB. These are compared to the default SRI build (i.e. without their smaller structures). I'm working on optimizing this even further.
Binary format via mmap is supported. Run ./build_binary to make one then pass the binary file name instead.
Currently, it assumes POSIX APIs for errno, sterror_r, open, close, mmap, munmap, ftruncate, fstat, and read. This is tested on Linux and the non-UNIX Mac OS X. I welcome submissions porting (via #ifdef) to other systems (e.g. Windows) but proudly have no machine on which to test it.
A brief note to Mac OS X users: your gcc is too old to recognize the pack pragma. The warning effectively means that, on 64-bit machines, the model will use 16 bytes instead of 12 bytes per n-gram of maximum order.
It does not depend on Boost or ICU. However, if you use Boost and/or ICU in the rest of your code, you should define USE_BOOST and/or USE_ICU in util/string_piece.hh. Defining USE_BOOST will let you hash StringPiece. Defining USE_ICU will use ICU's StringPiece to prevent a conflict with the one provided here. By the way, ICU's StringPiece is buggy and I reported this bug: http://bugs.icu-project.org/trac/ticket/7924 .
The recommend way to use this:
Copy the code and distribute with your decoder.
Set USE_ICU and USE_BOOST at the top of util/string_piece.hh as instructed above.
Look at compile.sh and reimplement using your build system.
Use either the interface in lm/ngram.hh or lm/virtual_interface.hh
Interface documentation is in comments of lm/virtual_interface.hh (including for lm/ngram.hh).
I recommend copying the code and distributing it with your decoder. However, please send improvements to me so that they can be integrated into the package.
Also included:
A wrapper to SRI with the same interface.
|
