Add rate limiting that respect retry-after and TorBox specific limits

[colinajd]

User description

• Added retry that uses Retry-After if available
• Added specific TorBox Hard Rate Limits

Ran into issues with adding loads of content to be downloaded by rdt-client, constantly seeing on active torrents that rate limit exceeded.

This has been completely vibe coded, but it built and I deployed a docker image at colinajd/rdt-client and am currently using it and all functionality seems to be working as expected.

Please review if you think this is a worthwhile attempt at resolving the specific rate limiting issue/ TorBox use case

---

PR Type

Enhancement

---

Description

• Implement TorBox-specific rate limiting with sliding window algorithm

• Add per-second and per-minute/hour limits for create torrent operations

• Migrate from Polly to Microsoft.Extensions.Http.Resilience for retry handling

• Support Retry-After header parsing for server-provided delay preferences

• Create separate HTTP clients for general and torrent creation requests

---

Diagram Walkthrough

flowchart LR
  A["HTTP Requests"] --> B["RD_CLIENT<br/>Default Retry Policy"]
  A --> C["TORBOX_CLIENT<br/>5 req/sec Rate Limit"]
  A --> D["TORBOX_CLIENT_CREATETORRENT<br/>10/min + 60/hour Limits"]
  B --> E["Exponential Backoff<br/>+ Retry-After Support"]
  C --> E
  D --> E

Loading

File Walkthrough

Relevant files

Enhancement

DiConfig.cs Implement TorBox rate limiting and retry strategy                server/RdtClient.Service/DiConfig.cs Added three static SlidingWindowRateLimiter instances for TorBox rate limiting (5/sec general, 10/min and 60/hour for create torrent) Replaced Polly HTTP resilience with Microsoft.Extensions.Http.Resilience framework Implemented custom retry strategy with Retry-After header parsing and exponential backoff with jitter Created three named HTTP clients: RD_CLIENT, TORBOX_CLIENT, and TORBOX_CLIENT_CREATETORRENT with appropriate resilience handlers Updated imports to use new resilience libraries and removed deprecated Polly.Extensions.Http +134/-11 TorBoxTorrentClient.cs Route torrent creation through rate-limited client              server/RdtClient.Service/Services/TorrentClients/TorBoxTorrentClient.cs Modified GetClient() method to accept optional client name parameter for selecting appropriate HTTP client Updated AddMagnet() to use TORBOX_CLIENT_CREATETORRENT client for torrent creation requests Updated AddFile() to use TORBOX_CLIENT_CREATETORRENT client for torrent creation requests Maintains backward compatibility by defaulting to TORBOX_CLIENT when no client specified +4/-4

---

[qodo-code-review]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🟢	No security concerns identified No security vulnerabilities detected by AI analysis. Human verification advised for critical code.
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Passed
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Passed
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Missing auditing: New torrent creation paths do not include any explicit logging or audit details for critical actions like adding magnets/files. Referred Code public async Task<String> AddMagnet(String magnetLink) { var user = await GetClient().User.GetAsync(true); var result = await GetClient(DiConfig.TORBOX_CLIENT_CREATETORRENT).Torrents.AddMagnetAsync(magnetLink, user.Data?.Settings?.SeedTorrents ?? 3, false); if (result.Error == "ACTIVE_LIMIT") { var magnetLinkInfo = MonoTorrent.MagnetLink.Parse(magnetLink); return magnetLinkInfo.InfoHashes.V1!.ToHex().ToLowerInvariant(); } return result.Data!.Hash!; } public async Task<String> AddFile(Byte[] bytes) { var user = await GetClient().User.GetAsync(true); var result = await GetClient(DiConfig.TORBOX_CLIENT_CREATETORRENT).Torrents.AddFileAsync(bytes, user.Data?.Settings?.SeedTorrents ?? 3); if (result.Error == "ACTIVE_LIMIT") ... (clipped 3 lines)
	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Retry edge cases: Retry strategy may compute negative or zero delays when Retry-After date is in the past and lacks explicit logging or handling for retry exhaustion and rate limit rejections. Referred Code if (args.Outcome.Result is HttpResponseMessage resp && resp.Headers.RetryAfter is { } ra) { if (ra.Delta is TimeSpan delta) { return new ValueTask<TimeSpan?>(delta); } if (ra.Date is DateTimeOffset when) { var delay = when - DateTimeOffset.UtcNow; return new ValueTask<TimeSpan?>(delay > TimeSpan.Zero ? delay : TimeSpan.Zero); } } // null => use the configured backoff (Delay/BackoffType/UseJitter) return new ValueTask<TimeSpan?>((TimeSpan?)null); }
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: External input checks: Methods accepting magnet links and file bytes rely on external API without visible validation or sanitation in the new paths for create-torrent operations. Referred Code public async Task<String> AddMagnet(String magnetLink) { var user = await GetClient().User.GetAsync(true); var result = await GetClient(DiConfig.TORBOX_CLIENT_CREATETORRENT).Torrents.AddMagnetAsync(magnetLink, user.Data?.Settings?.SeedTorrents ?? 3, false); if (result.Error == "ACTIVE_LIMIT") { var magnetLinkInfo = MonoTorrent.MagnetLink.Parse(magnetLink); return magnetLinkInfo.InfoHashes.V1!.ToHex().ToLowerInvariant(); } return result.Data!.Hash!; } public async Task<String> AddFile(Byte[] bytes) { var user = await GetClient().User.GetAsync(true); var result = await GetClient(DiConfig.TORBOX_CLIENT_CREATETORRENT).Torrents.AddFileAsync(bytes, user.Data?.Settings?.SeedTorrents ?? 3); if (result.Error == "ACTIVE_LIMIT")

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
High-level	Decouple provider-specific logic from core configuration The suggestion recommends moving provider-specific configurations, like the TorBox rate limiters, out of the shared DiConfig.cs file. This logic should be encapsulated within provider-specific modules to improve modularity and extensibility. Examples: server/RdtClient.Service/DiConfig.cs [29-60] private static readonly SlidingWindowRateLimiter TorboxPerSecondLimiter = new(new SlidingWindowRateLimiterOptions { PermitLimit = 5, Window = TimeSpan.FromSeconds(1), SegmentsPerWindow = 4, QueueLimit = 5, QueueProcessingOrder = QueueProcessingOrder.OldestFirst, AutoReplenishment = true }); ... (clipped 22 lines) Solution Walkthrough: Before: // In DiConfig.cs public static class DiConfig { // TorBox-specific rate limiters private static readonly SlidingWindowRateLimiter TorboxPerSecondLimiter = ...; private static readonly SlidingWindowRateLimiter TorboxCreateTorrentPerMinuteLimiter = ...; // ... public static void RegisterHttpClients(this IServiceCollection services) { // ... generic retry strategy ... // TorBox-specific client services.AddHttpClient(TORBOX_CLIENT, ...) .AddResilienceHandler("TorBox-Limits", builder => { builder.AddRateLimiter(args => TorboxPerSecondLimiter.AcquireAsync(...)); builder.AddRetry(...); }); // Another TorBox-specific client services.AddHttpClient(TORBOX_CLIENT_CREATETORRENT, ...) .AddResilienceHandler("TorBox-CreateTorrent-Limits", ...); } } After: // In DiConfig.cs public static class DiConfig { public static void RegisterHttpClients(this IServiceCollection services) { // Register clients from different providers services.RegisterRealDebridClient(); services.RegisterTorBoxClients(); // Encapsulated registration // Future providers can be added easily } } // In a new file, e.g., TorBoxServiceCollectionExtensions.cs public static class TorBoxServiceCollectionExtensions { public static IServiceCollection RegisterTorBoxClients(this IServiceCollection services) { // All TorBox-specific logic is encapsulated here var perSecondLimiter = new SlidingWindowRateLimiter(...); // ... other limiters services.AddHttpClient(DiConfig.TORBOX_CLIENT, ...) .AddResilienceHandler("TorBox-Limits", ...); services.AddHttpClient(DiConfig.TORBOX_CLIENT_CREATETORRENT, ...) .AddResilienceHandler("TorBox-CreateTorrent-Limits", ...); return services; } } Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a significant design issue where provider-specific logic for TorBox is hardcoded into the core DiConfig.cs, violating the Open/Closed Principle and hindering future extensibility.	Medium
Possible issue	Apply general rate limit to client The TORBOX_CLIENT_CREATETORRENT HTTP client is missing the general 5 requests/second rate limit. Add the TorboxPerSecondLimiter to its resilience handler to prevent exceeding the general API rate limit and avoid unnecessary retries. server/RdtClient.Service/DiConfig.cs [178-198] services.AddHttpClient(TORBOX_CLIENT_CREATETORRENT, httpClient => { httpClient.DefaultRequestHeaders.Add("User-Agent", UserAgent); }) .AddResilienceHandler("TorBox-CreateTorrent-Limits", (builder, ctx) => { + // General per-second limiter + builder.AddRateLimiter(new RateLimiterStrategyOptions + { + RateLimiter = args => TorboxPerSecondLimiter.AcquireAsync(1, args.Context.CancellationToken), + OnRejected = _ => default + }); + // First limiter: 10 per minute builder.AddRateLimiter(new RateLimiterStrategyOptions { RateLimiter = args => TorboxCreateTorrentPerMinuteLimiter.AcquireAsync(1, args.Context.CancellationToken), OnRejected = _ => default }); // Second limiter: 60 per hour builder.AddRateLimiter(new RateLimiterStrategyOptions { RateLimiter = args => TorboxCreateTorrentPerHourLimiter.AcquireAsync(1, args.Context.CancellationToken), OnRejected = _ => default }); builder.AddRetry(retryStrategy); }); Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that the TORBOX_CLIENT_CREATETORRENT is missing the general per-second rate limiter, which could lead to unnecessary 429 errors; applying it improves the robustness of the rate-limiting strategy.	Medium
More

[omgbeez]

I checked over this change, it won't actually solve the problem unfortunately.

Two main issues:

1. The Polly RateLimiter is fully client side, and so wouldn't correctly reflect server state when requests are being made to a TorBox account from another device/client, causing requests to still return an HTTP 429 exception which will likely have the same outcome as before this change.
2. The Polly RateLimiter itself returns its own exception when the client side Rate Limit has been reached, rejecting (not queuing) the request. This isn't handled in the change, so TorrentRunner.cs will simply get a generic exception and schedule a retry (effectively the same behaviour as before the change, when the server rate limit was hit).

Because the cooldown on createtorrent/createusenet can be in the range of an hour or more, using the HTTPClient to queue and handle that may not be the best way to go anyway unless also implementing a BulkHead policy based on the Max Concurrent downloads.

[omgbeez]

I've taken the Retry-After and resiliency parts of this change into #912 . Once asylumexp/TorBox.NET#5 is merged I'll look at retry handling the ACTIVE_LIMIT rates for TorBox.