Hercules Job System

A powerful background job processing system with prioritization, throttling, and scheduling capabilities.

Features

• Task Management: Define, queue, and execute background tasks
• Retry Mechanism: Automatic retries for failed tasks
• Scheduling: Run tasks immediately, at specific times, or on recurring schedules
• Prioritization: Different priority levels for tasks
• Throttling: Control concurrency of task execution
• State Management: Track task states (WAITING, READY, EXECUTING, COMPLETE, FAILED, THROTTLED)
• Namespaces: Group tasks by execution namespace
• Parent-Child Relationships: Tasks can trigger other tasks with priority inheritance
• Unique Keys: Prevent duplicate task execution
• Callbacks: Execute code on task completion or failure
• API Server: REST API for managing tasks and monitoring the system

Installation

npm install hercules-job-system

Configuration

Create a .env file in your project root:

# PostgreSQL configuration
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=hercules

# Redis configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0

# Hercules system configuration
HERCULES_DEFAULT_MAX_RETRIES=4
HERCULES_DEFAULT_TIMEOUT=900
HERCULES_DEFAULT_CONCURRENCY=100
HERCULES_DEFAULT_PRIORITY=DEFAULT
HERCULES_DEFAULT_CRITICALITY=MEDIUM
HERCULES_DEFAULT_EXECUTION_NAMESPACE=default
HERCULES_WORKER_COUNT=2
HERCULES_HEARTBEAT_INTERVAL=300000
HERCULES_CONTROLLER_INTERVAL=1000
HERCULES_ENABLE_THROTTLING=true
HERCULES_ENABLE_MONITORING=true
HERCULES_ENABLE_API=true
HERCULES_API_PORT=3000

Database Setup

Run the database setup script:

npx ts-node src/db/setup.ts

Usage

Basic Usage

import HerculesJobSystem, { hercules_task, Priority, Criticality } from 'hercules-job-system';

// Define a task using the decorator
class ExampleTasks {
  @hercules_task({
    priority: Priority.HIGH_PRIORITY,
    max_retries: 3,
    throttle: { concurrent: 5 },
    criticality: Criticality.HIGH
  })
  static async processItem(itemId: string): Promise<void> {
    console.log(`Processing item ${itemId}`);
    // Simulate some work
    await new Promise(resolve => setTimeout(resolve, 1000));
    console.log(`Item ${itemId} processed successfully`);
  }
}

// Create an instance of the Hercules job system
const herculesSystem = new HerculesJobSystem(['default'], 2);

// Register task handlers
herculesSystem.registerTask('ExampleTasks', 'processItem', ExampleTasks.processItem);

// Start the system
await herculesSystem.start();

// Schedule a task
await ExampleTasks.processItem.delay('item-1');

Recurring Tasks

@hercules_task({
  priority: Priority.NICE_TO_HAVE,
  max_retries: 1,
  schedule: {
    cron_expression: '0 12 * * *',
    timezone: 'UTC'
  },
  criticality: Criticality.LOW
})
static async generateDailyReport(): Promise<void> {
  console.log('Generating daily report');
  // Simulate generating a report
  await new Promise(resolve => setTimeout(resolve, 2000));
  console.log('Daily report generated');
}

Throttling

@hercules_task({
  priority: Priority.DEFAULT,
  max_retries: 2,
  throttle: { concurrent: 2 }, // Only 2 tasks can run concurrently
  criticality: Criticality.MEDIUM
})
static async sendEmail(to: string, subject: string, body: string): Promise<void> {
  console.log(`Sending email to ${to}`);
  // Simulate sending an email
  await new Promise(resolve => setTimeout(resolve, 500));
  console.log(`Email sent to ${to}`);
}

Unique Keys

@hercules_task({
  priority: Priority.HIGH_PRIORITY,
  unique_key: (args) => `process-order-${args[0]}`, // Unique key based on order ID
  unique_key_behavior: UniqueKeyBehavior.DROP_IF_EXISTS
})
static async processOrder(orderId: string): Promise<void> {
  console.log(`Processing order ${orderId}`);
  // Process the order
  console.log(`Order ${orderId} processed`);
}

API

The Hercules job system provides a REST API for managing tasks:

• GET /tasks: Get all tasks
• GET /tasks/:id: Get a task by ID
• POST /tasks: Create a task
• PUT /tasks/:id: Update a task
• DELETE /tasks/:id: Delete a task
• GET /states: Get task states
• GET /priorities: Get task priorities
• GET /criticalities: Get task criticalities
• GET /unique-key-behaviors: Get unique key behaviors

Architecture

The Hercules job system consists of the following components:

• Scheduler: Moves tasks between states (WAITING → READY → EXECUTING)
• Worker: Executes tasks
• API Server: REST API for managing tasks and monitoring the system
• Database: PostgreSQL for storing tasks, throttle configurations, and other data

License

MIT