GitHub - JoshuaMart/ScopesExtractor: A tool for monitoring bug bounty programs across multiple platforms to track scope changes.

A tool to automatically synchronize and track bug bounty program scopes from multiple platforms. Monitor new programs, scope changes, and receive Discord notifications for updates.

⚠️ Version 2.x Warning

This is version 2.x of ScopesExtractor which contains breaking changes from version 1.x. If you're upgrading from 1.x, please review the CHANGELOG.md for more informations

Features

🔄 Multi-Platform Support: YesWeHack, HackerOne, Intigriti, Bugcrowd
📊 Automatic Synchronization: Continuously monitor programs and detect scope changes
🔔 Discord Notifications: Get notified about new programs, scope changes, and removals
🗄️ SQLite Database: Persistent storage with historical change tracking
🌐 REST API: Query scopes and changes programmatically
🎯 Smart Scope Processing: Automatic validation and normalization with platform-specific rules

Installation

Prerequisites

Ruby >= 3.4.0
SQLite3
libcurl (for Typhoeus)

Configuration

Copy the environment template:
```
cp .env.example .env
```
Configure platform credentials in .env (email, password, API tokens, TOTP secrets)
Configure application settings in config/settings.yml (enable/disable platforms, Discord webhooks, etc.)

Using Docker (Recommended)

# Build the image
docker build -t scopes_extractor .

# Create named volume for database
docker volume create scopes_db

# Run with mounted config and named volume for database
docker run -v $(pwd)/config/settings.yml:/app/config/settings.yml:ro \
           -v $(pwd)/.env:/app/.env:ro \
           -v scopes_db:/app/db \
           scopes_extractor \
           bundle exec bin/scopes_extractor sync

Local Installation

# Install dependencies
bundle install

# Run migrations
bundle exec bin/scopes_extractor migrate

CLI Usage

Commands

Sync Programs

# Sync all enabled platforms
bundle exec bin/scopes_extractor sync

# Sync specific platform
bundle exec bin/scopes_extractor sync hackerone

# Verbose output
bundle exec bin/scopes_extractor sync -v
bundle exec bin/scopes_extractor sync yeswehack --verbose

Start API Server

# Start API server
bundle exec bin/scopes_extractor serve

# Custom port and bind address
bundle exec bin/scopes_extractor serve -p 8080 -b 127.0.0.1

# Enable auto-sync in background
bundle exec bin/scopes_extractor serve --sync

# Verbose logging
bundle exec bin/scopes_extractor serve -v

Database Management

# Run migrations
bundle exec bin/scopes_extractor migrate

# Cleanup old history entries
bundle exec bin/scopes_extractor cleanup

# Reset database (WARNING: deletes all data)
bundle exec bin/scopes_extractor reset
bundle exec bin/scopes_extractor reset --force  # Skip confirmation

Other Commands

# Display version
bundle exec bin/scopes_extractor version

# Show help
bundle exec bin/scopes_extractor help

Docker Usage

Docker Compose

Create a docker-compose.yml:

services:
  scopes_extractor:
    build: .
    container_name: scopes_extractor
    volumes:
      - ./config/settings.yml:/app/config/settings.yml:ro
      - ./.env:/app/.env:ro
      - scopes_db:/app/db  # Use named volume for database
    ports:
      - "4567:4567"
    command: bundle exec bin/scopes_extractor serve --sync
    restart: unless-stopped

volumes:
  scopes_db:  # Persistent database volume

Run with:

docker-compose up -d

Docker Run Examples

# Create named volume first
docker volume create scopes_db

# Sync once
docker run --rm \
           -v $(pwd)/config/settings.yml:/app/config/settings.yml:ro \
           -v $(pwd)/.env:/app/.env:ro \
           -v scopes_db:/app/db \
           scopes_extractor \
           bundle exec bin/scopes_extractor sync

# Start API server with auto-sync
docker run -d \
           -v $(pwd)/config/settings.yml:/app/config/settings.yml:ro \
           -v $(pwd)/.env:/app/.env:ro \
           -v scopes_db:/app/db \
           -p 4567:4567 \
           --name scopes_extractor \
           scopes_extractor \
           bundle exec bin/scopes_extractor serve --sync

# View logs
docker logs -f scopes_extractor

API Documentation

The REST API provides programmatic access to scopes and change history.

GET / - List all scopes

Query Parameters

Parameter	Type	Description
`platform`	string	Filter by platform name (e.g., `hackerone`, `bugcrowd`)
`type`	string	Filter by scope type (e.g., `web`, `mobile`, `api`)
`bounty`	boolean	Filter by bounty status (`true` or `false`)
`slug`	string	Filter by program slug
`values_only`	boolean	Return only scope values as array

Example Request

curl -H "X-API-KEY: your_api_key" "http://localhost:4567/?platform=hackerone&type=web&bounty=true"

Example Response

{
  "scopes": [
    {
      "slug": "example-program",
      "platform": "hackerone",
      "program_name": "Example Program",
      "bounty": true,
      "value": "*.example.com",
      "type": "web",
      "is_in_scope": true
    },
    {
      "slug": "example-program",
      "platform": "hackerone",
      "program_name": "Example Program",
      "bounty": true,
      "value": "api.example.com",
      "type": "web",
      "is_in_scope": true
    }
  ],
  "count": 2
}

Example Response (values_only=true)

[
  "*.example.com",
  "api.example.com"
]

GET /changes - Recent changes in history

Query Parameters

Parameter	Type	Description
`hours`	integer	Number of hours to look back (default: 24)
`platform`	string	Filter by platform name
`type`	string	Filter by event type (`new_program`, `removed_program`, `new_scope`, `removed_scope`)

Example Request

curl -H "X-API-KEY: your_api_key" "http://localhost:4567/changes?hours=48&platform=bugcrowd&type=new_scope"

Example Response

{
  "changes": [
    {
      "id": 123,
      "program_id": 45,
      "program_slug": "example-program",
      "platform_name": "bugcrowd",
      "event_type": "new_scope",
      "scope_value": "newapp.example.com",
      "scope_type": "web",
      "created_at": "2026-01-10T14:30:00Z"
    },
    {
      "id": 122,
      "program_id": 46,
      "program_slug": "another-program",
      "platform_name": "bugcrowd",
      "event_type": "new_scope",
      "scope_value": "*.another.com",
      "scope_type": "web",
      "created_at": "2026-01-10T12:15:00Z"
    }
  ],
  "count": 2
}

GET /wildcards - List all wildcard scopes

Query Parameters

Parameter	Type	Description
`platform`	string	Filter by platform name
`values_only`	boolean	Return only wildcard values as array

Example Request

curl -H "X-API-KEY: your_api_key" "http://localhost:4567/wildcards?platform=hackerone"

Example Response

{
  "wildcards": [
    {
      "slug": "example-program",
      "platform": "hackerone",
      "program_name": "Example Program",
      "bounty": true,
      "value": "*.example.com",
      "type": "web",
      "is_in_scope": true
    },
    {
      "slug": "another-program",
      "platform": "hackerone",
      "program_name": "Another Program",
      "bounty": false,
      "value": "*.another.org",
      "type": "web",
      "is_in_scope": true
    }
  ],
  "count": 2
}

Example Response (values_only=true)

[
  "*.example.com",
  "*.another.org"
]

GET /exclusions - List all excluded/ignored assets

Example Request

curl -H "X-API-KEY: your_api_key" "http://localhost:4567/exclusions"

Example Response

{
  "exclusions": [
    {
      "id": 1,
      "value": "admin.example.com",
      "reason": "Out of scope - admin panel",
      "created_at": "2026-01-09T10:00:00Z"
    },
    {
      "id": 2,
      "value": "internal.example.com",
      "reason": "Internal use only",
      "created_at": "2026-01-08T15:30:00Z"
    }
  ],
  "count": 2
}

Discord Notifications

Configure Discord webhooks to receive real-time notifications:

Notification Types

new_program: New bug bounty program discovered
removed_program: Program no longer available
new_scope: New scope added to a program
removed_scope: Scope removed from a program
ignored_asset: Asset failed validation and was ignored

Scope Type Filtering

Use new_scope_types to filter which scope types trigger notifications:

discord:
  webhooks:
    main:
      new_scope_types: ["web"]  # Only notify for web scopes
      # Or leave empty/null to notify for all types

Scope Processing

ScopesExtractor includes intelligent scope processing with automatic normalization and validation.

Auto-Heuristic Type Detection

Scopes are automatically categorized based on pattern matching, overriding platform-provided types when applicable:

Pattern	Detected Type	Example
GitHub/GitLab URLs	`source_code`	`https://github.com/user/repo`
Atlassian Marketplace	`source_code`	`https://marketplace.atlassian.com/apps/123`
App Store URLs	`mobile`	`https://apps.apple.com/app/id123`
Play Store URLs	`mobile`	`https://play.google.com/store/apps/details?id=com.app`
Chrome Web Store	`executable`	`https://chrome.google.com/webstore/detail/ext`
CIDR notation	`cidr`	`192.168.1.0/24`
Wildcard domains	`web`	`*.example.com`

Platform-Specific Normalization

Each platform has custom normalization rules to handle their scope formats:

YesWeHack

Expands multi-TLD patterns: example.{fr,com} → example.fr, example.com
Handles prefix patterns: {www,api}.example.com → www.example.com, api.example.com

HackerOne

Replaces .* with .com: example.* → example.com
Replaces .(TLD) with .com: example.(TLD) → example.com
Splits comma-separated values: domain1.com,domain2.com → domain1.com, domain2.com

Intigriti

Replaces <tld> with .com: *.example.<tld> → *.example.com
Splits slash-separated values: domain1.com / domain2.com → domain1.com, domain2.com

Bugcrowd

Extracts primary domain from dash-separated descriptions: example.com - Production → example.com

Global Normalization

Applied to all scopes regardless of platform:

Converts leading dots to wildcards: .example.com → *.example.com
Removes trailing slashes and wildcards: example.com/* → example.com
Downcases all values: Example.COM → example.com
Cleans up escaped characters and extra spaces

Validation Rules

Scopes are validated before being added to the database. Invalid scopes trigger ignored_asset notifications.

Rejected patterns:

Values without dots (unless IP addresses)
Multiple wildcards: *.xyz.*.example.com ❌
Invalid wildcard placement: example*.com ❌
Template placeholders: {id}, <identifier>, [name]
Descriptions in parentheses: example.com (production only)
Sentence punctuation: periods, commas, semicolons in unexpected positions
Values with spaces (except in URLs with query parameters)
Very short values (< 4 characters)
Hash symbols in domain portion (allowed in URL fragments)

Accepted patterns:

Standard domains: example.com ✅
Subdomains: api.example.com ✅
Wildcards: *.example.com ✅
URLs with protocols: https://example.com ✅
URLs with paths: https://example.com/api ✅
IP addresses: 192.168.1.1 ✅
CIDR ranges: 10.0.0.0/8 ✅

Development

Running Tests

# Run all tests
bundle exec rspec

# Run with coverage
bundle exec rspec --format documentation

# Run specific test file
bundle exec rspec spec/scopes_extractor/models/scope_spec.rb

Code Quality

# Run RuboCop
bundle exec rubocop

Project Structure

scopes_refactor/
├── bin/
│   └── scopes_extractor          # CLI executable
├── lib/
│   └── scopes_extractor/
│       ├── api.rb                # REST API server
│       ├── auto_sync.rb          # Background sync scheduler
│       ├── cli.rb                # Thor CLI commands
│       ├── config.rb             # Configuration loader
│       ├── database.rb           # Database connection & migrations
│       ├── diff_engine.rb        # Program diff & change detection
│       ├── http.rb               # HTTP client with cookie support
│       ├── normalizer.rb         # Scope value normalization
│       ├── sync_manager.rb       # Platform synchronization orchestration
│       ├── validator.rb          # Scope validation logic
│       ├── models/               # Dry-Struct models
│       ├── notifiers/            # Discord notifications
│       └── platforms/            # Platform-specific implementations
│           ├── base_platform.rb
│           ├── yeswehack/
│           ├── hackerone/
│           ├── intigriti/
│           ├── bugcrowd/
│           └── immunefi/
├── spec/                         # RSpec tests
├── config/
│   └── settings.yml              # Main configuration
├── Dockerfile
└── Gemfile

License

This project is licensed under the MIT License.

Name		Name	Last commit message	Last commit date
Latest commit History 391 Commits
.github/workflows		.github/workflows
bin		bin
config		config
db/migrations		db/migrations
lib		lib
spec		spec
.env.example		.env.example
.gitignore		.gitignore
.rspec		.rspec
.rubocop.yml		.rubocop.yml
CHANGELOG.md		CHANGELOG.md
Dockerfile		Dockerfile
Gemfile		Gemfile
Gemfile.lock		Gemfile.lock
LICENSE		LICENSE
README.md		README.md
docker-compose-dev.yml		docker-compose-dev.yml
docker-entrypoint.sh		docker-entrypoint.sh

License

JoshuaMart/ScopesExtractor

Folders and files

Latest commit

History

Repository files navigation

Features

Installation

Prerequisites

Configuration

Using Docker (Recommended)

Local Installation

CLI Usage

Commands

Sync Programs

Start API Server

Database Management

Other Commands

Docker Usage

Docker Compose

Docker Run Examples

API Documentation

Query Parameters

Example Request

Example Response

Example Response (values_only=true)

Query Parameters

Example Request

Example Response

Query Parameters

Example Request

Example Response

Example Response (values_only=true)

Example Request

Example Response

Discord Notifications

Notification Types

Scope Type Filtering

Scope Processing

Development

Running Tests

Code Quality

Project Structure

License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Uh oh!

Languages