ngram-merge

NAME

SYNOPSIS

ngram-merge [ -help ] [ -write outfile ] [ -float-counts ] \
	[ -- ] infile1 infile2 ...

DESCRIPTION

The input format consists of one N-gram count per line,

	word1 word2 ... wordn count

Each filename argument can be a plain ASCII count file, or a compressed file (name ending in .Z or .gz), or ``-'' to indicate stdin/stdout.

ngram-merge is recommended in cases where the full counts would far exceed available real memory. Although an arbitrary number of input count files is accepted, it is best to use the program as follows. First, partition the input text into the largest chunks so that ngram-count can run in real memory. Then merge the resulting sorted counts using ngram-merge pairwise, and continue doing so in a binary tree pattern until a single count file containing all N-grams remains. This procedure is automated by the make-batch-counts and merge-batch-counts scripts.

OPTIONS

Each filename argument can be an ASCII file, or a compressed file (name ending in .Z or .gz), or ``-'' to indicate stdin/stdout.

-help

Print option and usage summary.

-version

Print version information.

-write outfile

Write merged counts to outfile, instead of standard output.

-float-counts

Process counts as floating point numbers. By default counts are assumed to be unsigned integers.

--

Indicates the end of options, in case the first input filename begins with ``-''.

SEE ALSO

AUTHOR