[Feature] Alternative to Github API (rate limit hits too soon)

[RafaelJohn9]

Subcategory

feature-enhancement - Improving existing features

Description

• Github API rate limits are reaching too soon, despite being light operations. We may need an alternative and a sustainable solution to this
• We may have to use Github API only when its absolutely neccessary

[RafaelJohn9]

Github API offers 60 requests per hour for Unauthenticated Users and Up to 5000 requests per hour for authenticated users.

This does not favor the tool at all.

Mitigation Thoughts

Store files that rarely challenge inside the repo (high)

• Having the repo stored with a bulk of templates, with well known and maintained Dir structure will lead to the migration from github api to raw github user content. For both authenticated and unauthenticaed Users have the same rate limits

Respect the current rate window

• Read the X‑RateLimit‑Remaining and X‑RateLimit‑Reset headers on every call. If the remaining quota is low, queue new requests until the reset epoch.

Aggressive local caching

• Cache responses from API

Authenticate by default (low confidence)

• Prompt users for a Personal Access Token (PAT) on first run and securely persist it (e.g., in the OS keyring). Fall back to the unauthenticated budget only if no token is available.

[RafaelJohn9]

How we currently use the GitHub API.

• fetch and list various available templates
• Add templates
• Preview templates (License)
• Deal with Licenses

[RafaelJohn9]

Why we fetch licenses mainly via GitHub API

• Initially, we stored license files directly within the repository. However, we had concerns about the integrity and authenticity of those license files.

• To address this, we propose adding a --verify option that will validate the license content, whether it's local or part of the template repository, against a trusted source (e.g. SPDX).

• This verification is important because tampered or modified licenses—whether accidental or malicious—could lead to legally invalid or void licenses.

How we can mitigate the use of Github API in dealing with Licenses

• To fetch licenses we can use spdx either by using the raw github user content or going through spdx.org/licenses/<license.json>

Comparison if we use raw github user content or spdx JSON format

GitHub Raw vs SPDX.org License Data: Comparison Table

Feature / Criteria	GitHub Raw	SPDX.org
Where it lives	GitHub object store (global Fastly CDN). Same infra that serves every GitHub raw file.	The SPDX Foundation’s own web hosting (also on a CDN, but smaller footprint).
Data format	Complete license‑list archive in JSON, YAML, XML, RDF, and text. One giant licenses.json plus per‑license details/*.json.	Per‑license small JSON blob (≈ 3–5 KB) for each ID. No “all‑in‑one” file.
Typical access pattern	Download once, cache locally; or fetch specific detail files. Great for bulk tooling and offline packaging.	Fetch one license at a time from the web; better for lightweight, ad‑hoc look‑ups.
Latency / throughput	⚡ Fast — hits the same FASTLY POPs GitHub uses for source code. Large objects still stream quickly.	⚡ Fast for single‑file requests (static JSON, CDN‑cached); slower if you need dozens of IDs in quick succession (one HTTP round‑trip each).
Rate limits / quotas	Practically none for raw GETs; GitHub only throttles at abusive levels (> thousands req/s/IP) and there are no auth tokens needed.	Also effectively rate‑limit‑free, but more sensitive to scraping bursts (smaller infra, Cloudflare shields).
Scalability for bulk jobs	Excellent — grab the whole list once, parse locally, run millions of operations offline.	Poorer for bulk: N HTTP calls = licenses you need. You’d quickly overwhelm the endpoint if you fetched 500+ IDs in a loop.
Versioning / pinning	Can lock to a tag or commit SHA (e.g. v3.24). Perfect for reproducible builds.	No explicit versioning per JSON; you get “latest”. Harder to reproduce historical data.
Offline / air‑gapped	Perfect: just vendor the JSON files in your repo/container.	Must pre‑cache each license individually or lose access offline.
Update cadence	Merged instantly on SPDX list updates (because the repo is the canonical source).	Mirrors the same data, but can lag minutes‑to‑hours until site rebuild/deploy.
Failure blast radius	Tied to GitHub availability (rarely down; huge global infra).	Tied to SPDX site; smaller ops team, occasional maintenance windows.
Ideal use cases	• License scanners, SBOM generators, compliance CI/CD pipelines. • Build‑time embedding of full license set. • Situations needing deterministic snapshots.	• User‑facing dashboards showing a single license on demand. • Lightweight CLI helpers (spdx get MIT). • Educational tools linking directly to canonical SPDX URL.
Trade‑offs	Must download 3–6 MB to get the whole list (still tiny, but bigger).	Extra network round‑trips for each ID; no bulk file.
Winner for “scales the most”	GitHub Raw — bulk, offline, high‑volume safe.	—
Winner for “fastest one‑off lookup”	Tie (both are CDN‑cached static JSON), but spdx.org edges out if you only need a single ID once.

Conclusion

We go with raw github user content ⭐

[RafaelJohn9]

We can completely remove the use of GitHub API. by the use manifest.yml.

Having a good manifest.yml that can be accessed via raw github user content and would be cached locally after access, will solve multiple of our issues including listing templates, breaking down schemas from foreign repo , bash completions.

And honestly I'm not looking for another refactor dealing with licenses.

having a powerful manifest.yml may actually help keeping things at a minimum in the codebase. Though we may have to check the different approaches that we are to take

[RafaelJohn9]

Concluded approach

• minimize calls to github API.
• cache calls to github API.

[RafaelJohn9]

• Create manifest.yml for each template subdir inside our template folder.
• We will cache a list of outsourced repository locally (as we did with gitignore).
• Add tests to verify the integrity of the templates inside the repo by checking the manifest.yml. (handled by #112 )
• Change from Github API to use mostly raw githubuser content (each subcommand separated by its own commit),

[RafaelJohn9]

choosealicense.com seems like an interesting repo

[RafaelJohn9]

LIST COMMAND

gh-templates license list ✓
gh-templates license list --popular ✓
gh-templates license list --non-software ✓
gh-templates license list --include_deprecated ✓
gh-templates license list --search ✓