SML/NJ Benchmark Suite

This repository contains benchmark programs and supporting script for measuring the performance of the SML/NJ system. These include the "classic" benchmarks that have frequently been used to evaluate the performance of SML/NJ (and other SML systems) in the published literature. The original version of this suite was assembled by Lal George at Bell Laboratories in the late 1990s.

This version of the benchmark suite adds many additional programs, increases the problem sizes for some programs, and modernizes the code for others.

Running the Benchmarks

Each directory contains a script called runit. To invoke it, the shell variable SML must be set to the path for the SML/NJ compiler. 'runit' executed at the top level will create a file called LOG that contains the results of compiling and running each benchmark.

Executing

bin/run.sh -help

will show the command line options.

Benchmark Directory Structure

Each benchmark directory contains the following files (not all of which may be present):

• main.sml -- a SML file that defines the Main structure.

• sources.cm -- a CM file for building the benchmark program.

• additional SML source files for the benchmark.

• FILES -- an optional list of additional source files (other than main.sml) in compilation order. The bin/make-single-file.sh script uses this list to construct a single-file version of the program. This file is omitted for benchmarks that have main.sml as their only source file.

• ANSWER -- an optional text file containing the correct output from running the Main.testit function. Some benchmarks currently do not have test output; for these benchmarks, the ANSWER file is omitted.

• DATA -- an optional directory containing any input data files required by the benchmark program.

The Benchmark Programs

The following is a brief description of each benchmark program. The programs are listed by directory name in alphabetical order; for more detailed information, see the README.md file in each directory.

Directory	LOC	Short Description
barnes-hut	820	Hierarchical N-body solver (BROKEN)
black-scholes	127	European-style option pricing
boyer	912	(BROKEN)
cml-sieve	161	Message-passing implementation of Sieve of Eratosthenes
count-graphs	400
delta-blue	703	Delta-Blue incremental constraint solver (BROKEN)
dlx	1,887	(BROKEN)
fannkuchredux	xxx	(BROKEN)
fft	165
id-ray	xxx	A ray tracer ported from Id
kcfa	xxx	A reference implementation of kCFA (BROKEN)
knuth-bendix	433	Knuth-Bendix completion algorithm
lexgen	1,040	Lexical-analyzer generator
life	121	Conway's game of life
logic	322
mandelbrot	49	Mandelbrot-set computation
mandelbrot-rat	92	Mandelbrot-set computation using rationals
mazefun	180	Maze generator
mc-ray	736	Monte-Carlo ray tracer
minimax	167	Minimax game-tree search for Tic-Tac-Toe
mlyacc	5,572	LALR(1) parser generator
nbody	135	O(n^2) N-body solver
nucleic	2,919
pia	xxx	(BROKEN)
pingpong	135	Message-passing microbenchmark
plclub-ray	2,025	A raytracer from the ICFP'20 Programming Contest
ratio-regions	485	Image segmentation/contour finding
ray	xxx	A minimal ray tracer
regex	xxx	(BROKEN)
sat	53	Brute-force SAT solver
simple	760	A spherical fluid-dynamics program
smith-nf	371	Computes the Smith Normal Form
stream-sieve	57	Functional stream implementation of Sieve of Eratosthenes
tsp	302	A Traveling-Sales-Person solver
twenty-four	145	CPS-style solver for the 24 puzzle
vliw	2,939	A Very-Long-Instruction-Word instruction scheduler

Scripts

In addition to the run.sh script, there are a couple of other utility scripts in the bin directory.

cloc.sh

The cloc.sh script can be used to count the number of source lines in the benchmark programs. It uses the cloc program and reports the number of blank, comment, and code lines.

make-single-file.sh

The make-single-file.sh script is used to create a single compilation unit for the benchmark sources. It is used by the cloc.sh and run.sh