dfget support digest for directory

[pirDOL]

Feature request:

support digest for directory to verify the integrity of the downloaded file

Use case:

now:

--digest <algorithm>:<digest>

proposal:

--digest <algorithm>:<digest> # for backward compatibility
--digest file://<path> # new format, <path> is relative to download URL

/path/to/digest/file is in json
{
    "<URL1>": {"algorithm": "str", "digest": "str"}, 
    "<URL2>": {"algorithm": "str", "digest": "str"}, 
    ...
}

key design point:

1. Using relative path to store digest file makes digest and download file same source.
2. Using a file which contains multiple digests of each file in directory can avoid too long command line args.
3. Using url to locate each file and as digest file key makes it easier to get digest by url
4. The url of the digest itself can be also stored in the digest file, which is used to check the integrity of digest itself.

dfget pseudo code

# 1 download and check digest
if --digest file://; then
    digest_file_content = download(digest_url)
fi
# 2 parse digest
url2digest = json.unmarshal(digest_file_content)
# 3 check digest integrity
if url2digest[digest_file_path] != digest(digest_file_content)
    exit(1)
# 4 pass url2digest to DownloadRequest when download dir
entry_args.url = entry_url
entry_args.digest = url2digest[entry_url]
# 4 download directory
...

UI Example:

NA

[gaius-qi]

@pirDOL Here's a suggestion for implementing digest verification for directories:

Proposed CLI Option

--digest-manifest <path>

Digest Manifest Format

Use a format similar to sha256sum output, which is widely recognized and easy to generate/parse:

sha256:f1972cdafdcfe4d06288dfc31e49c86c877c4d530062d05860ec4460cf5697f3 ./file1.txt
sha256:0a2da6af39432ffaf2b8b2bf3516c54f9333c2286c9b4f3f6b4801712985dfa5 ./subdir/file2.txt
sha256:c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 /home/file3.txt

Key Benefits

1. Familiar Format: The sha256sum-style format is well-known and can be easily generated using standard tools
2. Relative/Absolute Path Support: Supports both relative paths (./file1.txt) and absolute paths (/home/file3.txt)
3. Simple Parsing: Each line contains <algorithm>:<digest> <filepath>, making it straightforward to parse
4. Backward Compatibility: The existing --digest <algorithm>: <digest> option remains unchanged for single file verification

Usage Example

# Download directory with digest verification
dfget -O /path/to/output --digest-manifest /path/to/checksums.txt https://example.com/files/

# The digest manifest file can be downloaded from the same source
dfget -O /tmp/checksums.txt https://example.com/files/checksums.txt
dfget -O /path/to/output --digest-manifest /tmp/checksums.txt https://example.com/files/