ytextractor

A powerful command-line tool to fetch video metadata, thumbnails, and download videos from YouTube channels using the YouTube Data API v3 and yt-dlp.

---

Features

• Comprehensive Data Fetching: Retrieves video metadata (title, description, published date, duration, thumbnails) for channels.
• Flexible Channel Identification: Supports identifying channels by their unique Channel ID (UC...), modern @handle, or legacy username.
• Local Data Storage: All channel and video metadata is stored persistently in a local SQLite database (yt.db).
• Organized Thumbnail Management: Downloads and saves thumbnails into structured folders (e.g., thumbnails/@channel_handle/).
• Video Downloading with Presets: Download videos in various quality presets (best, normal (up to 720p), small (up to 360p)).
• Structured Video Output: Videos are saved in an organized directory structure (e.g., videos/@channel_handle/) with detailed filenames (YYMMDD_video_name_video_duration_videoid_preset.mp4).
• In-line Download Progress: Displays real-time download progress on a single line in your terminal.
• Cutoff Date Filtering: When fetching, optionally exclude videos published before a specified date.
• Force Fetch/Redownload: Option to force re-fetching all metadata and re-downloading thumbnails.
• API Quota Monitoring: Includes commands to test your API key and get a "best-effort" API quota status check to help you manage your API usage.
• Detailed Logging: Tracks fetch operations, downloads, and errors in ytextractor.log. Logging level is configurable via environment variable.
• User-Friendly CLI: Provides straightforward commands via typer.
• Secure API Key Handling: Configurable API key via a .env file for secure environment variable management.
• Automated Versioning: Integrates commitizen for conventional commits and automated version bumping.
• Task Automation: Utilizes Taskfile.yml for simplified execution of common development and operational tasks.

---

Setup

Requirements

• Python 3.12+
• Poetry for robust dependency management.
• YouTube Data API v3 Key (requires a Google Cloud Project).
• FFmpeg: Required by yt-dlp for video merging and format conversion. Ensure it's installed and accessible in your system's PATH.

Installation

1. Clone or Download the Repository:

   git clone [https://github.com/your-username/ytextractor.git](https://github.com/your-username/ytextractor.git) # Replace with your repo URL
   cd ytextractor

2. Create a .env File: In the project root, create a file named .env and add your YouTube Data API Key. Optionally, you can set the logging level:

   YOUTUBE_API_KEY=YOUR_YOUTUBE_API_KEY_HERE
   YTEXTRACTOR_LOG_LEVEL=INFO # Optional: DEBUG, INFO, WARNING, ERROR, CRITICAL

3. Configure Poetry to Use Your Python Environment & Install Dependencies:

   a. If you want Poetry to manage its own virtual environment (recommended for simplicity, it creates a .venv folder in your project):

   poetry install

   b. If you prefer to use an existing Python virtual environment (e.g., managed by pyenv, conda, or python -m venv): First, activate your desired environment (e.g., source /path/to/my/venv/bin/activate). Then, tell Poetry to use it:

   poetry env use $(pyenv which python)
   poetry install

   This step (poetry env use) only needs to be done once per project unless you switch environments.

4. (Optional) Add Channels to channels.list: Create a file named channels.list in the project root. Add YouTube channel identifiers (Channel ID, @handle, or username) one per line. Lines starting with # will be ignored.

   # Example channels.list
   UC-lHJZR3Gqxm24_Vd_DKFwa # PewDiePie's channel ID
   @MrBeast # MrBeast's handle
   googledevelopers # A legacy username example
   

---

Usage

Once dependencies are installed and your API key is set up, you can run the tool two ways:

1. Activate the environment, and call ytextractor directly:

eval "$(poetry env activate)
ytextractor --help

2. Or run it via poetry:

poetry run python ytextractor.py <command>.

Note

If you choose option 2, replace all ytextractor commands below with poetry run python ytextractor.py

Fetch Videos Metadata and Thumbnails

Fetch videos metadata and thumbnails for all channels listed in channels.list:

ytextractor fetch

Fetch for a specific channel by ID, handle, or username:

ytextractor fetch UC-lHJZR3Gqxm24_Vd_DKFwa
ytextractor fetch @MrBeast

Force fetch all videos from the beginning (overwriting existing metadata and re-downloading thumbnails):

ytextractor fetch --force-all

Fetch videos published after a specific cutoff date (YYMMDD format):

ytextractor fetch --cutoff 231231 # Only videos published on or after Dec 31, 2023

You can combine options:

ytextractor fetch --force-all --cutoff 240101

Download Videos

Downloads videos based on the metadata already present in your database (yt.db). Videos will be saved in videos/@channel_handle/.

Download videos for all channels in channels.list using the default 'best' quality preset:

ytextractor fetch-videos

Download videos with a specific quality preset ('normal' for up to 720p, 'small' for up to 360p):

ytextractor fetch-videos --preset normal

Download a Single Video (Ad-Hoc)

Download a single video directly by its URL or ID, without interacting with the database. Videos will be saved to the current directory or a specified output directory.

Download a video using its URL or ID:

ytextractor download [https://www.youtube.com/watch?v=dQw4w9WgXcQ](https://www.youtube.com/watch?v=dQw4w9WgXcQ)
ytextractor download dQw4w9WgXcQ

Download a video with a specific quality preset:

ytextractor download dQw4w9WgXcQ --preset normal

Download to a specific output directory:

ytextractor download dQw4w9WgXcQ --dir /path/to/my/videos

Test API Key

Tests your YouTube Data API key by making a minimal request. This consumes 1 unit of your daily quota.

ytextractor test-api-key

Check API Quota Status

Provides an estimate of your remaining API quota. This also consumes 1 unit of quota.

ytextractor quota

Show Version

Displays the current version of the tool.

ytextractor --version

License

MIT License. See LICENSE for more information.