Implement autonomous recipe authoring system for RecipeKit

[Copilot]

Adds a fully autonomous system that generates, validates, and iteratively repairs RecipeKit scraping recipes from a single URL using AI-powered classification and evidence-based debugging.

System Architecture

Entry point: node scripts/autoRecipe.js --url=https://example.com

Four-phase autonomous workflow:

1. Classification - AI infers topic, canonicalizes folder (15+ mappings: film→movies, cooking→recipes)
2. Autocomplete - Generates search recipe, auto-generates tests, validates with engine
3. URL Recipe - Generates detail extraction, auto-generates tests, validates
4. Repair Loop - Classifies failures (6 types), collects evidence, AI fixes, retests (max 5×)

Implementation

Core orchestrator (scripts/autoRecipe.js, 27KB):

• Argument parsing, URL validation
• Domain extraction (handles subdomains, country-code TLDs: api.example.co.uk → example)
• Recipe generation workflows with validation
• Failure classification: SELECTOR_MISSING, JS_RENDERED, BOT_WALL, etc.
• Test generation (TypeScript following project conventions)

AI prompt templates (scripts/prompts/, 12KB):

• classify.md - Website topic classification with strict JSON
• author-autocomplete.md - Search recipe generation
• author-url.md - Detail page extraction
• fixer.md - Evidence-based recipe repair

Documentation (25KB):

• Architecture overview, usage, troubleshooting
• Complete workflow example with expected outputs
• Exact API references for Copilot SDK and agent-browser integration

Integration Points

Ready for integration (placeholder implementations with TODO markers):

class CopilotSession {
  async start() {
    // TODO: Initialize actual Copilot SDK
    // this.client = new CopilotClient();
    // this.session = await this.client.createSession({...});
  }
  
  async send(prompt) {
    // TODO: Send to Copilot, parse JSON response
    // return await this.session.send({ prompt });
  }
}

class WebProber {
  static async extractFingerprint(url) {
    // TODO: Use agent-browser for DOM extraction
    // await exec('agent-browser', ['open', url]);
    // return await exec('agent-browser', ['snapshot', '--json']);
  }
}

Working integrations:

• RecipeKit engine validation (spawns bun run Engine/engine.js)
• Test generation and file management
• Failure classification and repair loop logic

RecipeKit Engine Constraints

System generates recipes using supported commands only:

• Load: load, api_request
• Store: store, store_attribute, store_text, store_array, store_url, json_store_text
• Transform: regex, url_encode, replace

No interactive commands (click, fill, type) - workaround uses direct search URLs with query parameters.

Statistics

• 11 files, 2,571 lines, 64KB
• All automated tests passing (URL validation, domain extraction, file generation)
• Zero security vulnerabilities, consistent code style

Original prompt

Below is a clean, end-to-end rewrite of the full development plan, incorporating the exact Node.js Copilot SDK API you provided, plus a tight appendix with only the relevant agent-browser commands, Copilot SDK calls, and RecipeKit engine flags.

No hand-waving. No guessed APIs. This is implementable.

⸻

Autonomous Recipe Authoring for RecipeKit

(using agent-browser + Copilot SDK + Copilot CLI)

Objective

Build a fully autonomous system that, given a single URL, can:
1. Infer the main semantic topic of a website.
2. Decide where to store the recipe (folder + filename).
3. Automatically generate a valid RecipeKit recipe:
• First autocomplete_steps
• Then url_steps (detail)
4. Validate correctness by generating tests and running the RecipeKit engine.
5. Iteratively repair the recipe using:
• More web probing via agent-browser
• Copilot acting as author and fixer
6. Stop only when tests pass or a hard failure condition is reached.

Entrypoint:

node scripts/autoRecipe.js --url=https://some-website.com

No list type flags. No manual hints. Autonomy means autonomy.

⸻

System roles (non-negotiable)

agent-browser

The ground truth probe.
It navigates the website, reaches the correct UI state, and extracts minimal, structured evidence.

Copilot (via Copilot SDK + Copilot CLI)

The reasoning agent.
It:
• infers topic
• proposes folder
• writes RecipeKit steps
• fixes broken recipes based on test failures and new evidence

autoRecipe.js

The judge and orchestrator.
It:
• validates Copilot output
• enforces naming and schema rules
• writes files
• generates tests
• runs the RecipeKit engine
• controls the repair loop

Copilot proposes.
Your script disposes.

⸻

Phase 0: Repo structure (once)

Add the following structure:

recipes/
/
.json

tests/
generated/
/
.autocomplete.test.ts
.url.test.ts

scripts/
autoRecipe.js
prompts/
classify.md
author-autocomplete.md
author-url.md
fixer.md

Everything the agent generates must be committed or explicitly rejected.

⸻

Phase 1: Initial web probing and topic inference

Step 1.1 – Probe the website (agent-browser)

Given --url, the script:
1. Opens the page.
2. Takes a snapshot with refs.
3. Extracts:
• page title
• meta description
• main heading
• 1 representative “content card” if visible
• any JSON-LD structured data if present

This becomes the site fingerprint.

Do not dump full HTML.
Minimal, relevant evidence only.

⸻

Step 1.2 – Ask Copilot to classify and choose storage

Create a Copilot session using the real SDK:

const client = new CopilotClient();
await client.start();

const session = await client.createSession({
model: "gpt-5",
systemMessage: {
content: You are an autonomous agent that classifies websites and authors RecipeKit scraping recipes. Always respond with STRICT JSON. No prose.
}
});

Send the classification prompt:

await session.send({
prompt: `
Given the following website fingerprint, infer:

• the main topic
• a canonical folder name
• a confidence score
• a short rationale

Website fingerprint:
<fingerprint_json_here>

Respond with:
{
"topic": "...",
"folder": "...",
"confidence": 0.0,
"rationale": "..."
}
`
});

Wait for assistant.message and session.idle.

⸻

Step 1.3 – Validate and canonicalize folder (script-side)

Copilot does not get the final say.

Your script enforces:
• lowercase
• [a-z0-9-] only
• max 32 chars
• canonical mappings:
• film, cinema → movies
• novel, reading → books
• cooking, food → recipes
• shop, ecommerce → products

If invalid:
• send a fix request back to Copilot
• loop until valid or fail hard

Final storage path:

recipes//.json

⸻

Phase 2: Autocomplete recipe generation (closed loop)

Step 2.1 – Generate autocomplete_steps (Copilot author mode)

Provide Copilot with:
• site fingerprint
• snapshot evidence
• hints about search UI (if any)
• explicit instruction: use only supported RecipeKit commands

Copilot must respond with STRICT JSON:

{
"recipe": {
"title": "...",
"description": "...",
"engine_version": "1",
"url_available": ["..."],
"autocomplete_steps": [ ... ]
},
"testPlan": {
"queries": ["query1", "query2"]
}
}

No markdown. No explanation.

⸻

Step 2.2 – Write recipe and generate tests

Script writes:

recipes//.json

Then generates a test that runs:

bun run ./Engine/engine.js
--recipe recipes//.json
--type autocomplete
--input ""

Assertions:
• stdout parses as JSON
• result is an array
• length ≥ 3
• each item has:
• title
• URL

⸻

Step 2.3 – Run test and classify failures

If test fails, classify:
• selector missing
• JS-rendered content
• wrong URL pattern
• search flow incomplete
• bot wall / consent gate

This classification is done by the script, not Copilot.

⸻

Step 2.4 – Repair loop (autocomplete)

If more info is needed:
1. agent-brow...

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.