Skip to content

feat(api): add HTTP Range request support for streaming #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Zorlin wants to merge 9 commits into
base: main
Choose a base branch
from
Open

feat(api): add HTTP Range request support for streaming #82

Zorlin wants to merge 9 commits into from

Conversation

@Zorlin
Copy link

@Zorlin Zorlin commented Dec 26, 2025

Implements byte-range requests (RFC 7233) for file downloads:
- Add RangeStream wrapper for partial content streaming
- Parse Range header (e.g., "bytes=0-499", "bytes=500-")
- Return 206 Partial Content with Content-Range header
- Return 416 Range Not Satisfiable for invalid ranges
- Advertise Accept-Ranges: bytes on all file responses

  Enables seeking in media players for audio/video content.

  🤖 Generated with [Claude Code](https://claude.com/claude-code)

markspanbroek and others added 9 commits December 17, 2025 15:10
@markspanbroek
chore: update to nim 2.2.6
028d09f
@markspanbroek
fix(validation): do not use range type
54a7a7c 
This leads to problems serializing TOML, see also:
status-im/nim-serialization#104
@markspanbroek
fix(ci): remove nim version from nimble.lock
9cda557
@markspanbroek
fix(nimble): update groth16 to fix constantine dependency
018bac1
@markspanbroek
fix(nimble): update circom-witnessgen to fix constantine dependency
49688fc
@Zorlin @claude
feat(api): add directory manifest support for folder uploads
4688eab 
Implements directory manifests for organizing multiple files into a tree:

- New DirectoryCodec (0xCD04) for directory manifest CIDs
- DirectoryManifest type with entries, totalSize, filesCount
- Two-phase upload: files uploaded individually, then /directory endpoint
  finalizes them into a tree structure with POST JSON body
- HTML directory browsing with Archivist branding at /data/{cid}
- JSON directory listing when Accept header doesn't include text/html
- Path resolution within directories via /data/{cid}/path?p=subdir/file
- Auto-promotion of single-child root directories

API changes:
- POST /api/archivist/v1/directory - finalize directory from uploaded files
- GET /api/archivist/v1/data/{cid} - now serves HTML for directories
- Added MIME types: audio/mpeg, audio/flac, video/webm, etc.
- Relaxed filename validation to allow paths like "Album/track.mp3"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@Zorlin @claude
chore: ignore Claude Code config files
7d63ddb 
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@Zorlin @claude
feat(api): add HEAD support and fix traversal check for dotted filenames
6de464c 
- Add HEAD endpoint for /api/archivist/v1/data/{cid} to support
  metadata preflight checks (content-type, size) without full download
- Add HEAD endpoint for /api/archivist/v1/data/{cid}/network/stream
  to support jsmediatags ID3 tag reading
- Fix directory traversal check to allow ".." in filenames (e.g.
  "F.R.E.S.H..mp3") while still blocking actual traversal attacks

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@Zorlin @claude
feat(api): add HTTP Range request support for streaming
8a080d1 
Implements byte-range requests (RFC 7233) for file downloads:
- Add RangeStream wrapper for partial content streaming
- Parse Range header (e.g., "bytes=0-499", "bytes=500-")
- Return 206 Partial Content with Content-Range header
- Return 416 Range Not Satisfiable for invalid ranges
- Advertise Accept-Ranges: bytes on all file responses

Enables seeking in media players for audio/video content.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
github-actions[bot]
github-actions bot reviewed Dec 26, 2025
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

nph

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Lines 554 to 556 in 8a080d1

without subdirManifest =? (
await fetchDirectoryManifest(node.networkStore, subdirCid)
), err:

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Lines 559 to 564 in 8a080d1

entries.add(DirectoryEntry.new(
name = name,
cid = subdirCid,
size = subdirManifest.totalSize,
isDirectory = true,
))

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Lines 568 to 574 in 8a080d1

entries.add(DirectoryEntry.new(
name = f.name,
cid = f.cid,
size = f.size,
isDirectory = false,
mimetype = if f.mimetype.isSome: f.mimetype.unsafeGet() else: "",
))

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Lines 577 to 583 in 8a080d1

entries.sort(proc(a, b: DirectoryEntry): int =
if a.isDirectory and not b.isDirectory:
return -1
elif not a.isDirectory and b.isDirectory:
return 1
else:
return cmp(a.name, b.name)

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Lines 586 to 589 in 8a080d1

let dirManifest = DirectoryManifest.new(
entries = entries,
name = dirNode.name,
)

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Line 644 in 8a080d1

without directory =? (await fetchDirectoryManifest(node.networkStore, cidVal)), err:

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Line 656 in 8a080d1

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Lines 672 to 673 in 8a080d1

resp.setHeader("Content-Disposition",
"attachment; filename=\"" & manifest.filename.get() & "\"")

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Line 700 in 8a080d1

without directory =? (await fetchDirectoryManifest(node.networkStore, cidVal)), err:

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Line 729 in 8a080d1

return RestApiResponse.response($json, contentType = "application/json", headers = headers)

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Line 866 in 8a080d1

let pathParts = if pathStr.len > 0: pathStr.split('/') else: @[]

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Lines 870 to 872 in 8a080d1

return RestApiResponse.redirect(
Http307, "/api/archivist/v1/data/" & $cidVal
)

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Lines 896 to 898 in 8a080d1

return RestApiResponse.error(
Http404, "Path not found: " & part, headers = headers
)

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Line 916 in 8a080d1

await node.retrieveCid(foundEntry.cid, local = true, resp = resp, byteRange = byteRange)

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/api.nim

Line 925 in 8a080d1

without subDir =? (await fetchDirectoryManifest(node.networkStore, foundEntry.cid)), err:

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 22 in 8a080d1

const ArchivistLogoSvg* = """<svg version="1.0" xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 400 400" preserveAspectRatio="xMidYMid meet">

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 31 in 8a080d1

const DirectoryListingCss* = """

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 323 in 8a080d1

mime == "application/gzip" or mime == "application/x-7z-compressed":

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 359 in 8a080d1

dirName = if directory.name.len > 0: directory.name else: cidStr[0 ..< 12] & "..."

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 361 in 8a080d1

pathParts = if basePath.len > 0: basePath.strip(chars = {'/'}).split('/') else: @[]

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 363 in 8a080d1

var html = fmt"""<!DOCTYPE html>

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 389 in 8a080d1

html &= fmt""" <span class="separator">/</span>

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 393 in 8a080d1

html &= fmt""" </nav>

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 422 in 8a080d1

html &= fmt""" <div class="file-row">

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 435 in 8a080d1

let parentPath = if pathParts.len > 1:

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 439 in 8a080d1

html &= fmt""" <div class="file-row">

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 453 in 8a080d1

html &= """ <div class="empty-dir">This directory is empty</div>

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Lines 463 to 465 in 8a080d1

displayName = if entry.isDirectory: entryName & "/" else: entryName
html &= fmt""" <div class="file-row">

[nph] reported by reviewdog 🐶

archivist-node/archivist/rest/directoryhtml.nim

Line 477 in 8a080d1

html &= fmt""" </div>

[nph] reported by reviewdog 🐶

archivist-node/archivist/streams/rangestream.nim

Lines 27 to 32 in 8a080d1

type
RangeStream* = ref object of LPStream
source*: SeekableStream
rangeStart*: int # First byte of the range (inclusive)
rangeEnd*: int # Last byte of the range (inclusive)
bytesRemaining*: int # Bytes left to read in this range

[nph] reported by reviewdog 🐶

archivist-node/archivist/streams/rangestream.nim

Lines 40 to 43 in 8a080d1

T: type RangeStream,
source: SeekableStream,
rangeStart: int,
rangeEnd: int

[nph] reported by reviewdog 🐶

archivist-node/archivist/streams/rangestream.nim

Lines 50 to 53 in 8a080d1

source: source,
rangeStart: rangeStart,
rangeEnd: rangeEnd,
bytesRemaining: rangeLen

archivist/directorynode.nim
Comment on lines +24 to +25
ProtoBuffer, initProtoBuffer, getField, getRequiredRepeatedField,
write, finish, ProtoResult
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
ProtoBuffer, initProtoBuffer, getField, getRequiredRepeatedField,
write, finish, ProtoResult
ProtoBuffer, initProtoBuffer, getField, getRequiredRepeatedField, write, finish,
ProtoResult

archivist/directorynode.nim
Comment on lines +86 to +92
entries.add(DirectoryEntry(
name: entryName,
cid: entryCid,
size: size.NBytes,
isDirectory: isDir != 0,
mimetype: mimetype,
))
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
entries.add(DirectoryEntry(
name: entryName,
cid: entryCid,
size: size.NBytes,
isDirectory: isDir != 0,
mimetype: mimetype,
))
entries.add(
DirectoryEntry(
name: entryName,
cid: entryCid,
size: size.NBytes,
isDirectory: isDir != 0,
mimetype: mimetype,
)
)

archivist/directorynode.nim
Comment on lines +94 to +98
success DirectoryManifest(
entries: entries,
totalSize: totalSize.NBytes,
name: name,
)
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
success DirectoryManifest(
entries: entries,
totalSize: totalSize.NBytes,
name: name,
)
success DirectoryManifest(entries: entries, totalSize: totalSize.NBytes, name: name)

archivist/directorynode.nim
Comment on lines +140 to +142
cid = blk.cid,
entries = directory.entries.len,
totalSize = directory.totalSize
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
cid = blk.cid,
entries = directory.entries.len,
totalSize = directory.totalSize
cid = blk.cid, entries = directory.entries.len, totalSize = directory.totalSize

archivist/manifest/directory.nim
cid*: Cid
size*: NBytes
isDirectory*: bool
mimetype*: string # Empty string = not set
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
mimetype*: string # Empty string = not set
mimetype*: string # Empty string = not set

archivist/rest/api.nim
# Validate and normalize path
var normalPath = pathStr.replace("\\", "/")
while normalPath.len > 0 and normalPath[0] == '/':
normalPath = normalPath[1..^1]
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
normalPath = normalPath[1..^1]
normalPath = normalPath[1 ..^ 1]

archivist/rest/api.nim
for part in pathParts:
if part == "..":
return RestApiResponse.error(
Http400, "Invalid path (directory traversal not allowed): " & pathStr,
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
Http400, "Invalid path (directory traversal not allowed): " & pathStr,
Http400,
"Invalid path (directory traversal not allowed): " & pathStr,

archivist/rest/api.nim
Comment on lines +496 to +501
inputEntries.add(InputEntry(
path: normalPath,
cid: cidVal,
size: size,
mimetype: mimetypeOpt,
))
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
inputEntries.add(InputEntry(
path: normalPath,
cid: cidVal,
size: size,
mimetype: mimetypeOpt,
))
inputEntries.add(
InputEntry(path: normalPath, cid: cidVal, size: size, mimetype: mimetypeOpt)
)

archivist/rest/api.nim
Comment on lines +522 to +525
current.subdirs[part] = DirNode(
name: part,
subdirs: initTable[string, DirNode](),
)
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
current.subdirs[part] = DirNode(
name: part,
subdirs: initTable[string, DirNode](),
)
current.subdirs[part] =
DirNode(name: part, subdirs: initTable[string, DirNode]())

archivist/rest/api.nim
Comment on lines +529 to +534
current.files.add((
name: pathParts[^1],
cid: entry.cid,
size: entry.size,
mimetype: entry.mimetype,
))
Copy link
Contributor

@github-actions github-actions bot Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nph] reported by reviewdog 🐶

Suggested change
current.files.add((
name: pathParts[^1],
cid: entry.cid,
size: entry.size,
mimetype: entry.mimetype,
))
current.files.add(
(
name: pathParts[^1],
cid: entry.cid,
size: entry.size,
mimetype: entry.mimetype,
)
)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Zorlin @markspanbroek