StreamPressor is a stream compressor hardware generator written in the Chisel hardware construction language for evaluating various designs of streaming hardware compressors. The framework combines predefined hardware compressor primitives with user-defined primitives to generate Verilog code for simulation and integration with other hardware designs.
- Hardware Compression Pipeline: Complete X-ray data compression pipeline from .npy files to compressed output
- Bit Plane Compression: Advanced bit plane analysis and compression algorithms
- Lagrange Prediction: Hardware implementation of Lagrange-based prediction for data compression
- Variable-to-Fixed Conversion: V2F and F2V converters for efficient data packing
- Formal Verification: Comprehensive formal testing with bounded model checking
- Numpy Integration: Direct support for reading and processing .npy files via ScalaPy
- Bit Shuffling: Optimized bit shuffling algorithms for improved compression ratios
- Multi-format Support: Support for both 32-bit and 64-bit floating-point data
- Java SDK (8 or 11 recommended)
- sbt (see https://www.scala-sbt.org/download.html for installation)
- Verilator and z3 (for formal verification)
- Python 3 with numpy (for .npy file support)
- Linux environment is recommended
git clone https://github.com/kazutomo/StreamPressor.git
cd StreamPressorFor cc users, we provide an automated setup script that handles all dependencies and configuration:
# Make/Run the setup script executable
sh setup_streampressor.sh
The setup script will:
- Install Java 11, sbt, Verilator, Z3, and Python dependencies
- Fix Python library linking issues common in WSL
- Build the project and run initial tests
- Configure the environment automatically
Note: The script assumes you're in the project root directory. If you encounter any issues, you can still setup manually.
# Run all tests
sbt test
# Run with formal verification enabled
sbt "testOnly -- -DFORMAL=1"
# Run specific test suite
sbt "testOnly common.XRayCompressionPipelineSpec"# Generate Verilog for LPComp module
sbt 'runMain lpe.LPCompGen'
# List all available targets
sbt run# Compile and run tests
sbt test
# Generate Verilog codes for target modules
sbt run
# Run the compression ratio estimator
sbt 'runMain estimate.EstimateCR'
# Clean build artifacts
sbt cleanThe project includes a Makefile with convenient shortcuts:
# Run all tests
make test
# Run formal verification tests
make formal
# Run compression ratio estimator
make estimator
# Clean generated files
make cleanTo enable formal verification testing:
# Run all formal tests
sbt "testOnly -- -DFORMAL=1"
# Run specific formal test
sbt "testOnly common.LagrangePredFormalSpec -- -DFORMAL=1"StreamPressor/
├── src/
│ ├── main/scala/
│ │ ├── common/ # Core compression utilities
│ │ │ ├── BitPlaneCompressor.scala # Bit plane analysis and compression
│ │ │ ├── BitShuffle.scala # Bit shuffling algorithms
│ │ │ ├── BitShuffleUtils.scala # Bit shuffle utilities
│ │ │ ├── ClzParam.scala # Count leading zeros parameterized
│ │ │ ├── ConversionUtils.scala # V2F/F2V conversion utilities
│ │ │ ├── DataFeeder.scala # Data feeding and streaming
│ │ │ ├── F2VConv.scala # Fixed-to-Variable converter
│ │ │ ├── F2VConv.scala # Float-to-Vector converter
│ │ │ ├── Headers.scala # Header definitions
│ │ │ ├── IntegerizeFP.scala # Floating-point to integer conversion
│ │ │ ├── NumpyReaderScalaPy.scala # Numpy file reading via ScalaPy
│ │ │ ├── Utils.scala # Utility functions
│ │ │ ├── V2FConv.scala # Variable-to-Fixed converter
│ │ │ └── VFConv.scala # Vector-to-Float converter
│ │ ├── configs/ # Configuration modules
│ │ │ └── LPEComp.scala # Lagrange prediction compression config
│ │ ├── estimate/ # Compression ratio estimation
│ │ │ └── LPECompEstimateCR.scala # Compression ratio estimator
│ │ └── lpe/ # Lagrange prediction encoder/decoder
│ │ ├── LagrangePred.scala # Lagrange prediction core
│ │ └── LPEncoder.scala # Lagrange prediction encoder
│ └── test/scala/
│ ├── common/ # Core component tests
│ │ ├── BitShuffleSpec.scala # Bit shuffle tests
│ │ ├── ClzParamSpec.scala # Count leading zeros tests
│ │ ├── ConvTestPats.scala # Conversion test patterns
│ │ ├── DataFeederSpec.scala # Data feeder tests
│ │ ├── F2VConvSpec.scala # F2V converter tests
│ │ ├── IntegerizeFPSpec.scala # IntegerizeFP tests
│ │ ├── Misc.scala # Miscellaneous tests
│ │ ├── NumpyReaderScalaPySpec.scala # Numpy reader tests
│ │ ├── V2FConvSpec.scala # V2F converter tests
│ │ ├── V2FtoF2VSpec.scala # V2F/F2V loopback tests
│ │ ├── V2FtoF2VTest.scala # V2F/F2V integration tests
│ │ └── XRayCompressionPipelineSpec.scala # End-to-end pipeline tests
│ └── lpe/ # Lagrange prediction tests
│ ├── LagrangePredSpec.scala # Lagrange prediction tests
│ └── LPEncoderSpec.scala # LP encoder tests
├── test_data/ # Test data files
│ └── 25-trimmed.npy # X-ray test data (128KB)
├── misc/ # Miscellaneous files
│ └── swimplforcomparison/ # SWIMPL comparison tools
│ ├── disasmtest.c # Disassembly test
│ ├── Makefile # Build configuration
│ ├── measuretiming.c # Timing measurement
│ └── rdtsc.h # RDTSC header
├── .github/ # GitHub configuration
│ └── workflows/
│ └── test.yml # CI/CD workflow
├── .gitignore # Git ignore rules (342 lines)
├── .scalafmt.conf # Scala code formatting rules
├── build.sbt # SBT build configuration
├── LICENSE.txt # Argonne National Lab license
├── Makefile # Build shortcuts and targets
├── setup_streampressor.sh # Automated setup script for WSL
└── README.md # This documentation file
- Analyzes data sparsity across bit planes
- Eliminates zero bit planes for compression
- Provides detailed compression statistics
- Hardware implementation of Lagrange-based prediction
- Supports configurable coefficients
- Includes both encoder and decoder modules
- V2FConv: Converts variable-length data to fixed-size blocks
- F2VConv: Converts fixed-size blocks back to variable-length data
- Optimized for streaming data processing
- Direct .npy file reading via ScalaPy
- Support for chunked data processing
- Data statistics and analysis tools
The framework provides comprehensive compression analysis:
- Compression Ratio: Measures space savings achieved
- Bit Plane Sparsity: Analyzes data distribution across bit planes
- Zero Suppression: Tracks elimination of zero bit planes
- Processing Throughput: Hardware performance metrics
The project includes 38 comprehensive tests covering:
- Unit Tests: Individual component functionality
- Integration Tests: Complete pipeline testing
- Formal Verification: Bounded model checking
- Performance Tests: Compression ratio validation
common.*Spec- Core utility testslpe.*Spec- Lagrange prediction tests*FormalSpec- Formal verification testsXRayCompressionPipelineSpec- End-to-end pipeline tests
- Follow Scala and Chisel coding conventions
- Add tests for new functionality
- Update documentation for API changes
- Ensure all tests pass before submitting
This project is licensed under the Argonne National Laboratory Open Source License - see the LICENSE.txt file for details.
- Kazutomo Yoshii - Initial work - kazutomo@mcs.anl.gov
- Connor Bohannon - Documentation, testing, and X-ray compression features
- Based on research from T. Ueno et al., "Bandwidth Compression of Floating-Point Numerical Data Streams for FPGA-based High-Performance Computing"
- Developed at Argonne National Laboratory
- Built with Chisel hardware construction language
For questions and support:
- Open an issue on GitHub
- Contact: kazutomo@mcs.anl.gov