A TypeScript-based distributed recording processing pipeline simulator with queue management, concurrent processing, and ordered result merging.
- ✅ Asynchronous part processing
- ✅ Out-of-order part arrival handling
- ✅ Concurrent STT and FFMPEG processing
- ✅ Ordered merge logic
- ✅ Multiple session support
- ✅ CLI tools for monitoring
- ✅ Comprehensive logging
- ✅ >70% test coverage
- Node.js 18+ or 20+
- npm or yarn
git clone <repo-url>
cd recording-parts-queue
npm install
cp .env.example .envEdit .env file:
PORT=3000
MAX_CONCURRENT_PROCESSING=5
STT_MIN_DELAY_MS=5000
# ... see .env.example for all optionsnpm run dev # Development with auto-reload
npm start # Productionnpm run recorder # Single session
npm run recorder 3 # Three concurrent sessionsnpm run logs # View all logs
npm run logs:follow # Follow in real-timenpm run status # View all sessions
npm run status:detailed # Detailed view with previewsnpm run output # View merged results
npm run output:data # Text only
npm run output:audio # Audio only- Express Server: HTTP API for receiving parts
- Queue Service: p-queue for concurrent processing
- STT Service: Simulated speech-to-text (5-15s delay)
- FFMPEG Service: Simulated audio merging (5-15s delay)
- Session Manager: In-memory session state tracking
- Merge Logic: Ordered result merging
- Recorder Simulator: Test data generator
- Recorder sends parts → Server
- Server validates → Queue
- Queue processes via STT + FFMPEG (parallel)
- Results buffered by part index
- Merge consecutive completed parts
- Write to files when complete
Submit a recording part for processing.
Request:
{
"idRecord": "uuid",
"partIndex": 0,
"isLast": false
}Response:
{
"success": true,
"message": "Part queued for processing",
"data": {
"idRecord": "uuid",
"partIndex": 0,
"queuePosition": 3
}
}Get all recording sessions and their status.
Response:
{
"success": true,
"data": [
{
"idRecord": "uuid",
"totalParts": 5,
"processedParts": 5,
"isComplete": true,
"mergedText": "Part 0 text...\nPart 1 text...",
"mergedAudio": "Part 0 audio...\nPart 1 audio...",
"createdAt": "2025-10-05T10:00:00.000Z",
"completedAt": "2025-10-05T10:03:45.000Z"
}
]
}Health check endpoint.
Response:
{
"status": "healthy",
"uptime": 3600,
"timestamp": "2025-10-05T10:00:00.000Z"
}npm test # Run all tests
npm run test:watch # Watch mode
npm run test:coverage # With coverage reportCurrent coverage: >86% (branches, functions, lines, statements)
src/
├── types/ # TypeScript interfaces
├── utils/ # Logger, validation, helpers
├── services/ # STT, FFMPEG, Queue services
├── session/ # Session manager
├── server/ # Express app, routes, middleware
├── recorder/ # Simulator
└── cli/ # CLI commands
tests/
├── services/ # Unit tests
├── session/ # Session manager tests
└── integration/ # Full flow tests
Parts may arrive out of order, but merging only happens when consecutive parts are ready:
- Received: [0, 2, 3] → Merge [0]
- Received: [1] → Merge [1, 2, 3]
Session completes when:
- Total parts known (isLast received)
- All parts processed
- All parts merged (nextMergeIndex === totalParts)
Each part goes through STT and FFMPEG processing simultaneously:
- STT: Simulated speech-to-text processing
- FFMPEG: Simulated audio merging
- Both run in parallel for maximum efficiency
- Uses p-queue for concurrency control
- Configurable max concurrent processing (default: 5)
- Non-blocking API responses
- In-memory session state
npm run help # Show all available commands
npm run logs # View application logs
npm run logs:follow # Follow logs in real-time
npm run status # View session status
npm run status:detailed # Detailed session view
npm run output # View merged output files
npm run output:data # View text output only
npm run output:audio # View audio output only
npm run clean # Clean logs and output directories# Terminal 1: Start server
npm run dev
# Terminal 2: Send recordings
npm run recorder 2
# Terminal 3: Monitor processing
npm run logs:follow
# After completion
npm run status
npm run outputServer won't start:
- Check port is not in use:
lsof -i :3000 - Verify .env file exists
Parts not processing:
- Check logs:
npm run logs:follow - Verify server is running
- Check concurrency setting
Output files not created:
- Wait for all parts to complete
- Check OUTPUT_DIR permission
- Verify session completed:
npm run status
No sessions showing:
- Ensure server is running:
npm run dev - Check server URL in .env:
SERVER_URL=http://localhost:3000 - Verify API connectivity:
curl http://localhost:3000/health
- Create feature branch
- Write tests first
- Implement feature
- Ensure tests pass and coverage maintained
- Update documentation
- TypeScript strict mode
- No
anytypes - Functions <50 lines
- Files <300 lines
- ESLint compliant
See .env.example for all available configuration options:
- Server settings (PORT, NODE_ENV)
- Processing delays (STT_MIN_DELAY_MS, etc.)
- Concurrency limits (MAX_CONCURRENT_PROCESSING)
- Recorder simulation settings
- Logging configuration
- Output directory settings
- Handles 10+ concurrent sessions
- Up to 5 concurrent part processes
- API response time <100ms (excluding processing)
- No data loss during processing
- Idempotent handling for duplicate parts
- Graceful error handling and recovery
- Structured JSON logs via Pino
- Configurable log levels
- File and console output options
- Fisher-Yates Shuffle: For out-of-order part simulation
- Consecutive Merge: Only merge parts when sequence is complete
- Completion Buffer: Track processed parts for ordered assembly
- Express: Lightweight HTTP server
- p-queue: In-memory queue with concurrency control
- Pino: High-performance structured logging
- Zod: Runtime type validation
- UUID: Session identifier generation
- Axios: HTTP client for recorder simulator
MIT
🎉 Recording Parts Queue System - Production Ready!
This system successfully demonstrates distributed recording processing with queue management, concurrent processing, and ordered result merging - all implemented in TypeScript with comprehensive testing and monitoring capabilities.