diff --git a/CARD_ARBITRAGE_DESIGN.md b/CARD_ARBITRAGE_DESIGN.md new file mode 100644 index 0000000..078f474 --- /dev/null +++ b/CARD_ARBITRAGE_DESIGN.md @@ -0,0 +1,523 @@ +# Collectible Card Arbitrage App - Design Proposal + +## App Overview + +An automated arbitrage platform that monitors auction sites (e.g., Heritage Auctions, Sotheby's, local auction houses) and classifieds (Craigslist, Facebook Marketplace) for collectible card listings, compares prices against major marketplaces (eBay, TCGPlayer, Cardmarket), identifies profitable opportunities, and automates the purchase and resale process. + +## Core Architecture + +### System Components + +1. **Scraping Engine** - Monitors and extracts listing data from multiple sources +2. **Price Intelligence** - Aggregates and normalizes pricing data across marketplaces +3. **Arbitrage Calculator** - Determines profitability after fees, shipping, and risk factors +4. **Automation Engine** - Handles purchase and listing workflows +5. **User Dashboard** - Interface for monitoring, configuration, and manual oversight +6. **Notification System** - Alerts for opportunities, purchases, and issues + +--- + +## MVP Feature List + +### Phase 1: Core Monitoring & Analysis (Weeks 1-4) + +#### 1.1 Source Integration +- [ ] **Auction Site Scrapers** + - Heritage Auctions API/integration + - Local auction house websites (configurable URLs) + - Basic web scraping for non-API sources + +- [ ] **Classified Scrapers** + - Craigslist RSS feeds and web scraping + - Facebook Marketplace (via API or scraping) + - Basic filtering for collectible card categories + +- [ ] **Marketplace Price Aggregation** + - eBay Sold Listings API + - TCGPlayer API integration + - Cardmarket API (for European market) + - Manual price entry fallback + +#### 1.2 Card Identification & Matching +- [ ] **Card Recognition System** + - OCR for card images (optional, Phase 2) + - Text-based matching (card name, set, condition, variant) + - Fuzzy matching for typos and variations + - Set/edition identification (e.g., "Base Set", "1st Edition", "Shadowless") + +- [ ] **Condition Normalization** + - Standardize condition grades (PSA, BGS, CGC, Raw) + - Condition mapping between different grading systems + - Condition-based price adjustments + +#### 1.3 Price Analysis Engine +- [ ] **Market Price Calculation** + - Average sold prices (last 30/90 days) + - Price trend analysis (increasing/decreasing) + - Outlier detection and filtering + +- [ ] **Arbitrage Calculator** + - Source price + fees (auction fees, buyer's premium) + - Marketplace fees (eBay ~13%, TCGPlayer ~10%, etc.) + - Shipping costs (inbound and outbound) + - Estimated profit margin calculation + - Minimum profit threshold configuration + +#### 1.4 Basic Dashboard +- [ ] **Opportunity Feed** + - List of potential arbitrage opportunities + - Sortable by profit margin, price, urgency + - Card details, source, and calculated profit + +- [ ] **Manual Review Interface** + - Approve/reject opportunities + - View detailed price breakdown + - Historical opportunity tracking + +### Phase 2: Automation (Weeks 5-8) + +#### 2.1 Automated Purchasing +- [ ] **Auction Bidding** + - Automated bid placement (with max bid limits) + - Bid timing strategies (snipe vs. early bid) + - Budget management per user + +- [ ] **Direct Purchase** + - Automated "Buy It Now" purchases + - Payment processing integration (PayPal, Stripe) + - Purchase confirmation tracking + +#### 2.2 Automated Listing +- [ ] **Listing Creation** + - Auto-generate optimized titles and descriptions + - Image handling (download, optimize, upload) + - Category and tag assignment + - Pricing strategy (fixed price vs. auction) + +- [ ] **Multi-Platform Listing** + - eBay listing creation via API + - TCGPlayer listing creation + - Cross-platform inventory sync + +#### 2.3 Workflow Management +- [ ] **Order Tracking** + - Monitor purchase status + - Shipping tracking integration + - Receipt confirmation + +- [ ] **Inventory Management** + - Track cards in transit + - Track cards awaiting listing + - Track listed cards + - Track sold cards + +### Phase 3: Intelligence & Optimization (Weeks 9-12) + +#### 3.1 Advanced Analytics +- [ ] **Performance Dashboard** + - ROI tracking per card + - Success rate metrics + - Average profit margins + - Time-to-sale analysis + +- [ ] **Market Intelligence** + - Price trend predictions + - Seasonal demand patterns + - Card popularity scoring + +#### 3.2 Risk Management +- [ ] **Fraud Detection** + - Seller reputation checking + - Image authenticity verification (optional) + - Suspicious listing flagging + +- [ ] **Quality Control** + - Condition verification workflows + - Dispute handling system + - Return/refund automation + +#### 3.3 User Configuration +- [ ] **Custom Rules Engine** + - Set-specific filters (only certain sets) + - Price range filters + - Minimum profit thresholds + - Blacklist/whitelist cards + +- [ ] **Budget Controls** + - Daily/weekly/monthly spending limits + - Per-opportunity max bid limits + - Reserve fund management + +--- + +## Database Schema Extensions + +### New Models Required + +```python +# Card Catalog +class Card(SQLModel, table=True): + id: UUID + name: str # e.g., "Charizard" + set_name: str # e.g., "Base Set" + set_code: str # e.g., "BS1" + card_number: str | None + rarity: str | None + variant: str | None # "1st Edition", "Shadowless", etc. + created_at: datetime + +# Source Listings +class SourceListing(SQLModel, table=True): + id: UUID + source_type: str # "auction", "craigslist", "facebook" + source_id: str # External ID from source + source_url: str + card_id: UUID # FK to Card + title: str + description: str | None + current_price: Decimal + condition: str | None + images: list[str] # JSON array + end_time: datetime | None # For auctions + status: str # "active", "ended", "purchased", "expired" + scraped_at: datetime + last_updated: datetime + +# Marketplace Prices +class MarketplacePrice(SQLModel, table=True): + id: UUID + card_id: UUID # FK to Card + marketplace: str # "ebay", "tcgplayer", "cardmarket" + average_price: Decimal + min_price: Decimal + max_price: Decimal + recent_sales_count: int + price_trend: str # "increasing", "decreasing", "stable" + last_updated: datetime + +# Arbitrage Opportunities +class Opportunity(SQLModel, table=True): + id: UUID + source_listing_id: UUID # FK to SourceListing + card_id: UUID # FK to Card + source_price: Decimal + estimated_market_price: Decimal + estimated_profit: Decimal + profit_margin: Decimal # Percentage + fees_breakdown: dict # JSON + status: str # "pending", "approved", "purchased", "rejected", "expired" + created_at: datetime + reviewed_by: UUID | None # FK to User + +# Purchases +class Purchase(SQLModel, table=True): + id: UUID + opportunity_id: UUID # FK to Opportunity + user_id: UUID # FK to User + source_listing_id: UUID # FK to SourceListing + purchase_price: Decimal + fees: Decimal + total_cost: Decimal + purchase_date: datetime + status: str # "pending", "paid", "shipped", "received", "cancelled" + tracking_number: str | None + payment_method: str | None + +# Listings +class Listing(SQLModel, table=True): + id: UUID + purchase_id: UUID # FK to Purchase + user_id: UUID # FK to User + card_id: UUID # FK to Card + marketplace: str # "ebay", "tcgplayer", etc. + marketplace_listing_id: str # External ID + listing_url: str + list_price: Decimal + listing_type: str # "fixed", "auction" + status: str # "draft", "active", "sold", "ended", "cancelled" + created_at: datetime + sold_at: datetime | None + sold_price: Decimal | None + +# User Settings +class UserSettings(SQLModel, table=True): + id: UUID + user_id: UUID # FK to User (one-to-one) + min_profit_margin: Decimal # Minimum % profit required + max_purchase_price: Decimal + daily_spend_limit: Decimal + weekly_spend_limit: Decimal + monthly_spend_limit: Decimal + auto_approve_enabled: bool + preferred_marketplaces: list[str] # JSON array + card_set_filters: list[str] # JSON array + blacklisted_cards: list[UUID] # JSON array of Card IDs +``` + +--- + +## Implementation Requirements + +### Backend Extensions + +#### 1. New Dependencies +```toml +# Add to pyproject.toml +dependencies = [ + # Existing dependencies... + "beautifulsoup4>=4.12.0", # Web scraping + "selenium>=4.15.0", # Dynamic content scraping + "requests>=2.31.0", # HTTP requests + "lxml>=5.0.0", # HTML parsing + "pillow>=10.0.0", # Image processing + "python-dateutil>=2.8.0", # Date parsing + "fuzzywuzzy>=0.18.0", # Fuzzy string matching + "python-Levenshtein>=0.23.0", # String similarity + "celery>=5.3.0", # Background task processing + "redis>=5.0.0", # Celery broker + "ebaysdk>=2.1.5", # eBay API (or use official eBay REST API) + "stripe>=7.0.0", # Payment processing + "paypalrestsdk>=1.13.0", # PayPal integration +] +``` + +#### 2. New API Routes +``` +/api/v1/cards/ - Card catalog management +/api/v1/sources/ - Source listing management +/api/v1/opportunities/ - Arbitrage opportunities +/api/v1/purchases/ - Purchase tracking +/api/v1/listings/ - Listing management +/api/v1/marketplace-prices/ - Price data +/api/v1/scrapers/ - Scraper configuration and control +/api/v1/analytics/ - Performance metrics +/api/v1/settings/ - User settings +``` + +#### 3. Background Workers (Celery) +- **Scraper Workers**: Periodic scraping tasks for each source +- **Price Aggregation Workers**: Fetch and update marketplace prices +- **Opportunity Calculator**: Process new listings and calculate opportunities +- **Automation Workers**: Execute purchases and listings +- **Notification Workers**: Send alerts and emails + +#### 4. Scraper Architecture +```python +# Abstract base scraper +class BaseScraper: + def scrape_listings(self) -> list[SourceListing] + def normalize_card_data(self, raw_data: dict) -> dict + def extract_price(self, listing: dict) -> Decimal + +# Specific scrapers +class HeritageAuctionsScraper(BaseScraper) +class CraigslistScraper(BaseScraper) +class FacebookMarketplaceScraper(BaseScraper) +class EBayScraper(BaseScraper) # For price data +``` + +### Frontend Extensions + +#### 1. New Components +- `OpportunityCard.tsx` - Display arbitrage opportunities +- `OpportunityDetail.tsx` - Detailed opportunity view +- `PurchaseTracker.tsx` - Track purchase status +- `ListingManager.tsx` - Manage active listings +- `AnalyticsDashboard.tsx` - Performance metrics +- `ScraperConfig.tsx` - Configure scraping sources +- `UserSettings.tsx` - Extended settings for arbitrage rules +- `CardCatalog.tsx` - Browse and search card database + +#### 2. New Routes +``` +/opportunities - Opportunity feed +/opportunities/:id - Opportunity detail +/purchases - Purchase tracking +/listings - Active listings +/analytics - Performance dashboard +/settings/arbitrage - Arbitrage configuration +/cards - Card catalog +``` + +#### 3. Real-time Updates +- WebSocket integration for live opportunity updates +- Real-time purchase/listing status updates +- Live price updates + +### Infrastructure Requirements + +#### 1. Additional Services +- **Redis**: Celery broker and caching +- **Celery Workers**: Separate containers for background tasks +- **Proxy/VPN Service**: For scraping (to avoid IP bans) +- **Image Storage**: S3 or similar for card images +- **OCR Service**: Optional, for card image recognition + +#### 2. External API Accounts Needed +- eBay Developer Account (for API access) +- TCGPlayer API access +- Cardmarket API access +- PayPal Business Account +- Stripe Account +- Optional: Google Cloud Vision API (for OCR) + +#### 3. Legal & Compliance +- Terms of Service for automated purchasing +- Rate limiting to respect source websites +- Robots.txt compliance +- Data retention policies +- User consent for automated transactions + +--- + +## Technical Challenges & Solutions + +### Challenge 1: Web Scraping Reliability +**Problem**: Websites change structure, implement anti-bot measures +**Solution**: +- Robust error handling and retry logic +- Multiple scraping strategies (API > RSS > Scraping) +- Monitoring and alerting for scraper failures +- Fallback to manual entry + +### Challenge 2: Card Matching Accuracy +**Problem**: Same card listed with different names/formats +**Solution**: +- Fuzzy string matching algorithms +- Card database with aliases +- Machine learning for card recognition (Phase 2+) +- Manual review queue for uncertain matches + +### Challenge 3: Price Volatility +**Problem**: Prices change rapidly, opportunities expire +**Solution**: +- Real-time price updates +- Caching with short TTL +- Priority queue for time-sensitive opportunities +- Configurable refresh intervals + +### Challenge 4: Automation Risks +**Problem**: Automated purchases can go wrong (wrong card, damaged, fraud) +**Solution**: +- Manual approval workflow (MVP) +- Gradual automation with user-defined rules +- Fraud detection heuristics +- Dispute handling system +- Insurance/escrow for high-value purchases + +### Challenge 5: Rate Limiting & Bans +**Problem**: Aggressive scraping can get IP banned +**Solution**: +- Respectful rate limiting +- Proxy rotation +- User-agent rotation +- API usage where available +- Distributed scraping architecture + +--- + +## MVP Scope Reduction + +For a true MVP, consider starting with: + +1. **Manual Opportunity Discovery** + - Scrapers find opportunities + - User manually reviews and approves + - User manually completes purchase + - App tracks the purchase and suggests listing price + +2. **Single Marketplace Focus** + - Start with eBay only for both price data and listings + - Expand to other marketplaces later + +3. **Limited Automation** + - No automated purchasing in MVP + - Automated price monitoring and opportunity alerts only + - Manual listing creation with suggested prices + +4. **Basic Card Matching** + - Exact name matching first + - Manual matching for uncertain cases + - Expand to fuzzy matching in Phase 2 + +--- + +## Suggested App Names + +1. **CardFlip** - Simple, action-oriented +2. **ArbitrageCards** - Descriptive, SEO-friendly +3. **CardScout** - Suggests discovery and hunting +4. **FlipBot** - Playful, suggests automation +5. **CardArb** - Short, industry term +6. **ProfitPoke** - Pokemon reference, fun +7. **CardHunt** - Suggests finding opportunities +8. **MarketFlip** - Clear value proposition +9. **CardFlow** - Suggests smooth operations +10. **ArbitrageAI** - Emphasizes intelligence +11. **CardVault** - Suggests collection and value +12. **FlipEngine** - Technical, powerful +13. **CardSense** - Suggests smart decisions +14. **ProfitPulse** - Suggests real-time monitoring +15. **CardWise** - Suggests intelligence and wisdom + +**Top Recommendations:** +- **CardFlip** - Memorable, clear value prop +- **CardScout** - Professional, suggests discovery +- **FlipBot** - Modern, suggests automation + +--- + +## Development Timeline Estimate + +### MVP (12 weeks) +- Weeks 1-2: Database schema, basic models +- Weeks 3-4: Scraper infrastructure (1-2 sources) +- Weeks 5-6: Price aggregation (eBay API) +- Weeks 7-8: Opportunity calculation engine +- Weeks 9-10: Dashboard and UI +- Weeks 11-12: Testing, polish, deployment + +### Phase 2 (8 weeks) +- Weeks 13-14: Automated listing creation +- Weeks 15-16: Purchase tracking system +- Weeks 17-18: Additional marketplace integrations +- Weeks 19-20: Advanced analytics + +### Phase 3 (8 weeks) +- Weeks 21-22: Automated purchasing (with safeguards) +- Weeks 23-24: Multi-marketplace listing +- Weeks 25-26: Advanced matching and ML +- Weeks 27-28: Optimization and scaling + +--- + +## Success Metrics + +- **Opportunity Discovery Rate**: Cards found per day +- **Match Accuracy**: % of correctly identified cards +- **Price Accuracy**: % of opportunities with accurate profit calculations +- **User Engagement**: Opportunities reviewed per user +- **Conversion Rate**: % of opportunities that become purchases +- **ROI**: Average profit margin on completed flips +- **Time to Sale**: Average days from purchase to sale + +--- + +## Risk Considerations + +1. **Legal**: Ensure compliance with terms of service of scraped sites +2. **Financial**: Users risk money on automated purchases +3. **Technical**: Scrapers break when sites change +4. **Market**: Card prices can drop, eliminating profit +5. **Competition**: Other arbitrage bots may compete for same opportunities + +--- + +## Next Steps + +1. Validate market demand and user interest +2. Start with MVP scope (manual review, single marketplace) +3. Build scraper infrastructure for 1-2 sources +4. Integrate eBay API for price data +5. Create basic opportunity feed +6. Iterate based on user feedback diff --git a/IMPLEMENTATION_ROADMAP.md b/IMPLEMENTATION_ROADMAP.md new file mode 100644 index 0000000..99141b4 --- /dev/null +++ b/IMPLEMENTATION_ROADMAP.md @@ -0,0 +1,472 @@ +# Implementation Roadmap: Card Arbitrage App + +## Current Stack Analysis + +### Existing Infrastructure ✅ +- FastAPI backend with SQLModel ORM +- PostgreSQL database with Alembic migrations +- React frontend with TanStack Router +- User authentication system +- Basic CRUD operations +- Docker containerization +- CI/CD workflows + +### What Needs to Be Added + +--- + +## Phase 1: Database & Models (Week 1-2) + +### 1.1 Create New Database Models + +**File: `backend/app/models.py`** +- Extend existing models with: + - `Card` model (card catalog) + - `SourceListing` model (scraped listings) + - `MarketplacePrice` model (price data) + - `Opportunity` model (arbitrage opportunities) + - `Purchase` model (track purchases) + - `Listing` model (track listings) + - `UserSettings` model (arbitrage preferences) + +**Action Items:** +```bash +# Create new migration +cd backend +alembic revision --autogenerate -m "add_arbitrage_models" +alembic upgrade head +``` + +### 1.2 Update Existing Models + +**File: `backend/app/models.py`** +- Add relationships to `User` model: + - `purchases: list[Purchase]` + - `listings: list[Listing]` + - `opportunities: list[Opportunity]` + - `settings: UserSettings` (one-to-one) + +--- + +## Phase 2: Backend API Routes (Week 3-4) + +### 2.1 Create New Route Files + +**New Files:** +- `backend/app/api/routes/cards.py` - Card catalog endpoints +- `backend/app/api/routes/sources.py` - Source listing management +- `backend/app/api/routes/opportunities.py` - Opportunity feed +- `backend/app/api/routes/purchases.py` - Purchase tracking +- `backend/app/api/routes/listings.py` - Listing management +- `backend/app/api/routes/marketplace.py` - Price aggregation +- `backend/app/api/routes/scrapers.py` - Scraper control +- `backend/app/api/routes/analytics.py` - Performance metrics + +**File: `backend/app/api/main.py`** +- Register new routers: +```python +from app.api.routes import cards, sources, opportunities, purchases, listings, marketplace, scrapers, analytics + +api_router.include_router(cards.router, prefix="/cards", tags=["cards"]) +api_router.include_router(sources.router, prefix="/sources", tags=["sources"]) +api_router.include_router(opportunities.router, prefix="/opportunities", tags=["opportunities"]) +# ... etc +``` + +### 2.2 Update Existing Routes + +**File: `backend/app/api/routes/users.py`** +- Add endpoint to get/update user arbitrage settings + +--- + +## Phase 3: Scraper Infrastructure (Week 5-6) + +### 3.1 Install Dependencies + +**File: `backend/pyproject.toml`** +```toml +dependencies = [ + # ... existing dependencies + "beautifulsoup4>=4.12.0", + "selenium>=4.15.0", + "requests>=2.31.0", + "lxml>=5.0.0", + "python-dateutil>=2.8.0", + "fuzzywuzzy>=0.18.0", + "python-Levenshtein>=0.23.0", +] +``` + +**Action:** +```bash +cd backend +uv sync +``` + +### 3.2 Create Scraper Module + +**New Directory: `backend/app/scrapers/`** +- `__init__.py` +- `base.py` - Base scraper class +- `heritage_auctions.py` - Heritage Auctions scraper +- `craigslist.py` - Craigslist scraper +- `facebook_marketplace.py` - Facebook Marketplace scraper +- `utils.py` - Scraper utilities + +**File: `backend/app/scrapers/base.py`** +```python +from abc import ABC, abstractmethod +from app.models import SourceListing + +class BaseScraper(ABC): + @abstractmethod + async def scrape_listings(self) -> list[SourceListing]: + pass + + @abstractmethod + def normalize_card_data(self, raw_data: dict) -> dict: + pass +``` + +### 3.3 Create Scraper Service + +**File: `backend/app/services/scraper_service.py`** +- Orchestrates all scrapers +- Handles scheduling +- Manages scraper state + +--- + +## Phase 4: Background Tasks (Week 7) + +### 4.1 Install Celery & Redis + +**File: `backend/pyproject.toml`** +```toml +dependencies = [ + # ... existing + "celery>=5.3.0", + "redis>=5.0.0", +] +``` + +**File: `backend/docker-compose.yml`** +- Add Redis service: +```yaml +services: + redis: + image: redis:7-alpine + ports: + - "6379:6379" +``` + +### 4.2 Create Celery App + +**New File: `backend/app/core/celery_app.py`** +```python +from celery import Celery +from app.core.config import settings + +celery_app = Celery( + "app", + broker=f"redis://redis:6379/0", + backend=f"redis://redis:6379/0", +) +``` + +### 4.3 Create Background Tasks + +**New Directory: `backend/app/workers/`** +- `__init__.py` +- `scraper_tasks.py` - Scraping tasks +- `price_tasks.py` - Price aggregation tasks +- `opportunity_tasks.py` - Opportunity calculation tasks + +**Example Task:** +```python +from app.core.celery_app import celery_app + +@celery_app.task +def scrape_heritage_auctions(): + # Scraping logic + pass +``` + +### 4.4 Update Docker Compose + +**File: `backend/docker-compose.yml`** +- Add Celery worker service: +```yaml +services: + celery-worker: + build: . + command: celery -A app.core.celery_app worker --loglevel=info + depends_on: + - redis + - db +``` + +--- + +## Phase 5: Price Intelligence (Week 8) + +### 5.1 eBay API Integration + +**File: `backend/pyproject.toml`** +```toml +dependencies = [ + # ... existing + "ebaysdk>=2.1.5", # Or use official eBay REST API +] +``` + +**New File: `backend/app/services/marketplace_service.py`** +- eBay API client +- Price fetching logic +- Price normalization + +### 5.2 Price Aggregation Service + +**File: `backend/app/services/price_service.py`** +- Aggregates prices from multiple sources +- Calculates averages, trends +- Updates MarketplacePrice records + +--- + +## Phase 6: Arbitrage Engine (Week 9) + +### 6.1 Opportunity Calculator + +**New File: `backend/app/services/arbitrage_service.py`** +```python +class ArbitrageService: + def calculate_opportunity(self, source_listing: SourceListing) -> Opportunity: + # Calculate profit margins + # Factor in fees, shipping, etc. + pass +``` + +### 6.2 Fee Calculator + +**New File: `backend/app/services/fee_calculator.py`** +- Calculate auction fees +- Calculate marketplace fees +- Calculate shipping costs +- Calculate total profit + +--- + +## Phase 7: Frontend Components (Week 10-11) + +### 7.1 Generate API Client + +**Action:** +```bash +cd frontend +npm run generate-client +``` + +### 7.2 Create New Components + +**New Files:** +- `frontend/src/components/Opportunities/OpportunityCard.tsx` +- `frontend/src/components/Opportunities/OpportunityDetail.tsx` +- `frontend/src/components/Opportunities/OpportunityFeed.tsx` +- `frontend/src/components/Purchases/PurchaseTracker.tsx` +- `frontend/src/components/Listings/ListingManager.tsx` +- `frontend/src/components/Analytics/AnalyticsDashboard.tsx` +- `frontend/src/components/Cards/CardCatalog.tsx` +- `frontend/src/components/Settings/ArbitrageSettings.tsx` + +### 7.3 Create New Routes + +**New Files:** +- `frontend/src/routes/opportunities.tsx` +- `frontend/src/routes/opportunities.$id.tsx` +- `frontend/src/routes/purchases.tsx` +- `frontend/src/routes/listings.tsx` +- `frontend/src/routes/analytics.tsx` +- `frontend/src/routes/cards.tsx` + +**File: `frontend/src/routes/_layout.tsx`** +- Add navigation links for new routes + +### 7.4 Update Settings Route + +**File: `frontend/src/routes/settings.tsx`** +- Add arbitrage settings section + +--- + +## Phase 8: Real-time Updates (Week 12) + +### 8.1 WebSocket Integration + +**File: `backend/pyproject.toml`** +```toml +dependencies = [ + # ... existing + "websockets>=12.0", +] +``` + +**New File: `backend/app/api/websocket.py`** +- WebSocket endpoint for real-time updates + +**File: `frontend/src/hooks/useWebSocket.ts`** +- React hook for WebSocket connection + +--- + +## Configuration Updates + +### Environment Variables + +**File: `.env`** +```bash +# Existing variables... + +# Scraping +SCRAPER_USER_AGENT=CardFlip/1.0 +SCRAPER_RATE_LIMIT_DELAY=2 # seconds between requests + +# eBay API +EBAY_APP_ID=your_ebay_app_id +EBAY_DEV_ID=your_ebay_dev_id +EBAY_CERT_ID=your_ebay_cert_id +EBAY_SANDBOX=true + +# Celery +CELERY_BROKER_URL=redis://redis:6379/0 +CELERY_RESULT_BACKEND=redis://redis:6379/0 + +# Redis +REDIS_URL=redis://redis:6379/0 + +# Arbitrage Settings +DEFAULT_MIN_PROFIT_MARGIN=20 # percentage +DEFAULT_MAX_PURCHASE_PRICE=1000 # dollars +``` + +### Settings Model + +**File: `backend/app/core/config.py`** +- Add new settings fields: +```python +class Settings(BaseSettings): + # ... existing settings + + # Scraping + SCRAPER_USER_AGENT: str = "CardFlip/1.0" + SCRAPER_RATE_LIMIT_DELAY: int = 2 + + # eBay + EBAY_APP_ID: str | None = None + EBAY_DEV_ID: str | None = None + EBAY_CERT_ID: str | None = None + EBAY_SANDBOX: bool = True + + # Celery + CELERY_BROKER_URL: str = "redis://redis:6379/0" + CELERY_RESULT_BACKEND: str = "redis://redis:6379/0" +``` + +--- + +## Testing Strategy + +### Backend Tests + +**New Directory: `backend/tests/`** +- `test_scrapers.py` - Scraper unit tests +- `test_arbitrage.py` - Arbitrage calculation tests +- `test_marketplace.py` - Marketplace API tests +- `test_opportunities.py` - Opportunity service tests + +### Frontend Tests + +**New Directory: `frontend/src/__tests__/`** +- Component tests for new UI components +- Integration tests for workflows + +--- + +## Deployment Considerations + +### 1. Additional Infrastructure +- Redis instance (managed or containerized) +- Celery worker instances (can scale horizontally) +- Proxy/VPN service for scraping (optional) + +### 2. Monitoring +- Scraper health monitoring +- Opportunity discovery rate +- API rate limit tracking +- Error alerting + +### 3. Rate Limiting +- Implement rate limiting for scrapers +- Respect robots.txt +- Implement exponential backoff + +--- + +## Migration Checklist + +- [ ] Create database models +- [ ] Run Alembic migrations +- [ ] Install new Python dependencies +- [ ] Create scraper infrastructure +- [ ] Set up Celery and Redis +- [ ] Create background tasks +- [ ] Integrate eBay API +- [ ] Build arbitrage calculation engine +- [ ] Create API routes +- [ ] Generate frontend API client +- [ ] Build frontend components +- [ ] Create new routes +- [ ] Add WebSocket support +- [ ] Update environment variables +- [ ] Write tests +- [ ] Update Docker Compose +- [ ] Deploy to staging +- [ ] Test end-to-end workflows +- [ ] Deploy to production + +--- + +## Quick Start Commands + +```bash +# Backend setup +cd backend +uv sync +alembic revision --autogenerate -m "add_arbitrage_models" +alembic upgrade head + +# Frontend setup +cd frontend +npm install +npm run generate-client + +# Run with Docker +docker-compose up -d +docker-compose up celery-worker # In separate terminal +``` + +--- + +## Estimated Effort + +- **Backend Development**: 6-8 weeks +- **Frontend Development**: 3-4 weeks +- **Integration & Testing**: 2-3 weeks +- **Total MVP**: 12 weeks (3 months) + +This assumes: +- 1-2 developers +- Starting with manual review workflow (no full automation) +- 2-3 source integrations +- eBay as primary marketplace diff --git a/OPPORTUNITY_MODEL_EXPLANATION.md b/OPPORTUNITY_MODEL_EXPLANATION.md new file mode 100644 index 0000000..a55cbf9 --- /dev/null +++ b/OPPORTUNITY_MODEL_EXPLANATION.md @@ -0,0 +1,379 @@ +# Opportunity Model - Detailed Explanation + +## Overview + +The `Opportunity` model is the **core entity** that represents a potential arbitrage deal - a card that can be purchased at a lower price from one source (auction site, Craigslist, etc.) and resold at a higher price on a marketplace (eBay, TCGPlayer, etc.) for profit. + +Think of it as a **"deal alert"** that the system has identified and calculated to be profitable. + +--- + +## Model Structure + +```python +class Opportunity(SQLModel, table=True): + id: UUID # Unique identifier + source_listing_id: UUID # Links to the original listing + card_id: UUID # Links to the card catalog entry + source_price: Decimal # Price to buy the card + estimated_market_price: Decimal # Expected resale price + estimated_profit: Decimal # Profit after all costs + profit_margin: Decimal # Profit as percentage + fees_breakdown: dict # Detailed cost breakdown (JSON) + status: str # Current state in workflow + created_at: datetime # When opportunity was discovered + reviewed_by: UUID | None # User who reviewed it (if any) +``` + +--- + +## Field-by-Field Breakdown + +### 1. **`id: UUID`** +- Unique identifier for the opportunity +- Generated automatically when created +- Used to reference the opportunity throughout the system + +### 2. **`source_listing_id: UUID`** (Foreign Key) +- **Links to**: `SourceListing` model +- **Purpose**: References the original listing where the card is being sold +- **Example**: Links to a Craigslist ad or Heritage Auctions listing +- **Why it matters**: Allows you to see the original listing details, images, seller info + +### 3. **`card_id: UUID`** (Foreign Key) +- **Links to**: `Card` model (card catalog) +- **Purpose**: Identifies which specific card this opportunity is for +- **Example**: Links to "Charizard - Base Set - 1st Edition" +- **Why it matters**: Enables matching against marketplace prices for the same card + +### 4. **`source_price: Decimal`** +- **What it is**: The price you would pay to buy the card from the source +- **Example**: `$150.00` (current bid on auction) +- **Note**: This is the "buy" price, not including fees yet + +### 5. **`estimated_market_price: Decimal`** +- **What it is**: The expected price you can sell the card for on marketplaces +- **How it's calculated**: + - Aggregated from recent sales on eBay, TCGPlayer, etc. + - Takes into account condition, variant, and market trends +- **Example**: `$250.00` (average recent sales price) + +### 6. **`estimated_profit: Decimal`** +- **What it is**: The net profit after ALL costs +- **Calculation**: + ``` + estimated_profit = estimated_market_price + - source_price + - source_fees (auction fees, buyer's premium) + - marketplace_fees (eBay ~13%, TCGPlayer ~10%) + - shipping_costs (inbound + outbound) + ``` +- **Example**: `$250 - $150 - $15 - $32.50 - $10 = $42.50` + +### 7. **`profit_margin: Decimal`** +- **What it is**: Profit as a percentage of total cost +- **Calculation**: `(estimated_profit / total_cost) * 100` +- **Example**: `($42.50 / $207.50) * 100 = 20.5%` +- **Why it matters**: Helps compare opportunities regardless of card price + +### 8. **`fees_breakdown: dict`** (JSON) +- **What it is**: Detailed breakdown of all costs +- **Structure**: + ```json + { + "source_price": 150.00, + "source_fees": { + "buyers_premium": 10.00, + "processing_fee": 5.00 + }, + "marketplace_fees": { + "ebay_final_value": 32.50, + "payment_processing": 7.50 + }, + "shipping": { + "inbound": 5.00, + "outbound": 5.00 + }, + "total_cost": 207.50, + "estimated_revenue": 250.00 + } + ``` +- **Why it matters**: Transparency - users can see exactly where costs come from + +### 9. **`status: str`** +- **What it is**: Current state in the opportunity lifecycle +- **Possible values**: + - `"pending"` - Just discovered, awaiting review + - `"approved"` - User approved, ready to purchase + - `"purchased"` - Card has been bought + - `"rejected"` - User declined the opportunity + - `"expired"` - Listing ended or opportunity no longer valid +- **Why it matters**: Tracks the opportunity through the entire workflow + +### 10. **`created_at: datetime`** +- **When**: Timestamp when the opportunity was first discovered +- **Why it matters**: + - Shows how fresh the opportunity is + - Helps prioritize time-sensitive deals (auctions ending soon) + +### 11. **`reviewed_by: UUID | None`** (Foreign Key, Optional) +- **Links to**: `User` model +- **What it is**: Which user reviewed/approved this opportunity +- **Null if**: Opportunity hasn't been reviewed yet, or was auto-approved +- **Why it matters**: Audit trail and accountability + +--- + +## How Opportunities Are Created + +### Step-by-Step Process + +1. **Scraper finds a listing** + - Scraper monitors auction sites, Craigslist, etc. + - Finds a new listing: "Charizard Base Set - $150" + - Creates a `SourceListing` record + +2. **Card matching** + - System tries to match the listing to a `Card` in the catalog + - Uses fuzzy matching on card name, set, condition + - Links `SourceListing.card_id` to the matched `Card` + +3. **Price lookup** + - System queries `MarketplacePrice` records for this card + - Gets average sold prices from eBay, TCGPlayer, etc. + - Calculates `estimated_market_price` + +4. **Profit calculation** + - Arbitrage service calculates: + - Source price + fees + - Marketplace fees + - Shipping costs + - Net profit and margin + +5. **Filtering** + - Checks if profit margin meets user's minimum threshold + - Checks if price is within user's budget limits + - Applies user's custom filters (card sets, etc.) + +6. **Opportunity creation** + - If profitable and passes filters → Creates `Opportunity` record + - Status set to `"pending"` + - User gets notified + +--- + +## Opportunity Lifecycle + +``` +┌─────────────┐ +│ DISCOVERED │ ← Scraper finds listing, calculates profit +└──────┬──────┘ + │ + ▼ +┌─────────────┐ +│ PENDING │ ← Awaiting user review +└──────┬──────┘ + │ + ├─── User Reviews ───┐ + │ │ + ▼ ▼ +┌─────────────┐ ┌─────────────┐ +│ APPROVED │ │ REJECTED │ ← User declines +└──────┬──────┘ └─────────────┘ + │ + ▼ +┌─────────────┐ +│ PURCHASED │ ← Purchase made, creates Purchase record +└──────┬──────┘ + │ + ▼ +┌─────────────┐ +│ LISTED │ ← Card listed for sale (via Listing model) +└─────────────┘ +``` + +### Status Transitions + +- **pending → approved**: User clicks "Approve" or auto-approval rules match +- **pending → rejected**: User clicks "Reject" +- **pending → expired**: Listing ended, auction closed, or price changed +- **approved → purchased**: Purchase completed (creates `Purchase` record) +- **approved → expired**: Opportunity expired before purchase +- **purchased**: Terminal state (tracked via `Purchase` model after this) + +--- + +## Relationships with Other Models + +### 1. **SourceListing** (One-to-One) +```python +opportunity.source_listing # Get the original listing +``` +- Contains: Original listing URL, images, seller info, auction end time +- Used for: Viewing listing details, making purchase + +### 2. **Card** (Many-to-One) +```python +opportunity.card # Get card catalog entry +``` +- Contains: Card name, set, rarity, variant +- Used for: Matching, price lookups, display + +### 3. **User** (Many-to-One, Optional) +```python +opportunity.reviewed_by_user # Get reviewer +``` +- Contains: User who reviewed/approved +- Used for: Audit trail, user-specific opportunities + +### 4. **Purchase** (One-to-One, Optional) +```python +opportunity.purchase # Get purchase record (if purchased) +``` +- Created when: Opportunity is purchased +- Contains: Actual purchase price, tracking, status + +### 5. **MarketplacePrice** (Indirect, via Card) +```python +opportunity.card.marketplace_prices # Get price data +``` +- Used for: Calculating `estimated_market_price` + +--- + +## Example Scenario + +### Real-World Flow + +1. **Scraper discovers listing**: + - Heritage Auctions listing: "PSA 10 Charizard Base Set" + - Current bid: $1,500 + - Auction ends in 2 days + +2. **System matches card**: + - Matches to `Card`: "Charizard - Base Set - PSA 10" + - Links `source_listing.card_id` + +3. **Price lookup**: + - `MarketplacePrice` shows: + - eBay average: $2,200 + - Recent sales: $2,100, $2,300, $2,000 + - Sets `estimated_market_price = $2,200` + +4. **Profit calculation**: + ``` + Source price: $1,500.00 + Buyer's premium: $150.00 (10%) + Processing fee: $25.00 + eBay fees: $286.00 (13% of $2,200) + Shipping (in/out): $20.00 + ────────────────────────────── + Total cost: $1,981.00 + Estimated revenue: $2,200.00 + Estimated profit: $219.00 + Profit margin: 11.1% + ``` + +5. **Opportunity created**: + ```python + Opportunity( + source_listing_id=listing_123, + card_id=charizard_card_id, + source_price=1500.00, + estimated_market_price=2200.00, + estimated_profit=219.00, + profit_margin=11.1, + fees_breakdown={...}, + status="pending" + ) + ``` + +6. **User sees opportunity**: + - Dashboard shows: "PSA 10 Charizard - $219 profit (11.1%)" + - User reviews details, sees it's a good deal + - Clicks "Approve" → status → `"approved"` + +7. **Purchase**: + - System (or user) places bid/purchase + - Creates `Purchase` record + - Opportunity status → `"purchased"` + +--- + +## Query Patterns + +### Common Queries + +```python +# Get all pending opportunities for a user +pending = session.query(Opportunity).filter( + Opportunity.status == "pending" +).all() + +# Get opportunities above profit threshold +profitable = session.query(Opportunity).filter( + Opportunity.profit_margin >= 20.0, + Opportunity.status == "pending" +).all() + +# Get opportunities for specific card +charizard_ops = session.query(Opportunity).join(Card).filter( + Card.name == "Charizard" +).all() + +# Get opportunities expiring soon (auctions) +urgent = session.query(Opportunity).join(SourceListing).filter( + SourceListing.end_time < datetime.now() + timedelta(hours=24), + Opportunity.status == "pending" +).all() +``` + +--- + +## Key Design Decisions + +### 1. **Why separate from SourceListing?** +- **SourceListing** = Raw data from scrapers +- **Opportunity** = Calculated, filtered, user-facing deals +- Allows multiple opportunities from same listing (if price changes) +- Enables user-specific filtering and preferences + +### 2. **Why store calculated values?** +- Performance: Don't recalculate on every query +- History: Track what profit was estimated at discovery time +- Audit: See what the system thought vs. actual results + +### 3. **Why JSON for fees_breakdown?** +- Flexible: Different sources have different fee structures +- Detailed: Users want transparency +- Extensible: Can add new fee types without schema changes + +### 4. **Why status field?** +- Simple state machine +- Easy to filter and query +- Clear workflow progression +- Can add more states later if needed + +--- + +## Future Enhancements + +### Potential Additions + +1. **Priority score**: Algorithmic ranking of opportunities +2. **Confidence level**: How certain the system is about the match +3. **Time sensitivity**: Hours until auction ends +4. **Competition level**: How many other buyers are interested +5. **Historical performance**: Track if similar opportunities were profitable +6. **Auto-approval rules**: User-defined criteria for automatic approval + +--- + +## Summary + +The `Opportunity` model is the **bridge** between: +- **Discovery** (scraped listings) +- **Analysis** (price calculations) +- **Action** (purchases and listings) + +It represents a **calculated, profitable deal** that users can review, approve, and act upon. It's the central entity that makes the arbitrage app work by connecting all the pieces together.