Skip to content

Conversation

@elliotsayes
Copy link

A few problems I encountered that lead me to this particular solution:

  • When downloading in chunks via HTTP Range header, the arweave.net gateway will eventually rate limit your requests, even with large chunks. Gateway then does exponential backoff which can seriously impact download time.
  • Naively wrapping a stream result in WebWorkers and Isolates causes errors
  • dio streams are a lie - it downloads a buffer and wraps it in Stream.value
  • package:http on web still use XHR requests, also without real stream support: Migrate to the fetch API dart-lang/http#595

So to get around all of these issues:

  • Use a single HTTP request to prevent rate limiting
  • Use fetch on web (using fetch_client library instead of ardrive-http.ts to handle the more complex js/dart interop)
  • Use the low level HttpClient on mobile which just works

I've confirmed that the memory usage on web is stable. Not sure how to confirm on mobile

Caveats of this method:

  • Streamed responses do not run in a WebWorker or Dart Isolate
  • Lack of flow control in fetch. Seems that this cannot be fixed: Support for flow control? Zekfad/fetch_client#3
    • This means the possibility of unbounded memory usage, but only in circumstances where the download speed exceeds the save/decrypt speed. In practice this should be very rare (decrypt speed is 100s of MiB/s on my M1 air, disk speed even faster). Could crop up when saving to a slow flash drive or SD card? It may pay to warn users about this.

@thiagocarvalhodev thiagocarvalhodev self-assigned this Mar 21, 2023
@thiagocarvalhodev thiagocarvalhodev added the enhancement New feature or request label Mar 21, 2023
@karlprieb
Copy link
Contributor

Another thing I noticed is that the code is not retrying using retryDelay and retryStatusCodes.
As you may know, the gateway can limit you or even return a 5xx error temporarily and that's why we retry.

karlprieb
karlprieb previously approved these changes Mar 23, 2023
@karlprieb karlprieb self-requested a review March 24, 2023 13:59
Copy link

@matibat matibat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!

environment:
sdk: '>=2.18.5 <3.0.0'
flutter: 3.3.9
flutter: 3.7.0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO for @matibat: check what are the downsides of bumping the version for localizations

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just FYR already fixed a bunch of localisation errors in the ArDrive-web PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants