Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true

########################################
# Java
########################################
[*.java]
indent_style = space
indent_size = 4
max_line_length = off

########################################
# XML (POM files, Spring XML, misc)
########################################
[*.xml]
indent_style = space
indent_size = 2

########################################
# JSON
########################################
[*.json]
indent_style = space
indent_size = 2

########################################
# YAML (docker-compose, GitHub actions, configs)
# YAML SHOULD NEVER USE TABS — many parsers reject them
########################################
[*.{yml,yaml}]
indent_style = space
indent_size = 2


########################################
# Markdown
# Keep indentation small; used mainly for lists and fenced blocks.
########################################
[*.md]
indent_style = space
indent_size = 2
trim_trailing_whitespace = false # needed to preserve Markdown hard line breaks

########################################
# Properties files
# Key-value pairs; indenting rarely matters but 2 spaces is common.
########################################
[*.properties]
indent_style = space
indent_size = 2

########################################
# HTML (simple default)
########################################
[*.html]
indent_style = space
indent_size = 2
1 change: 1 addition & 0 deletions .git-blame-ignore-revs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
cabb1a4746c49db58102833a858f4152141a4c7f
39 changes: 26 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,41 @@

This is a standalone project providing the functionality to implement "streaming" up- and downloads.

It is primarily meant to be used within the Step product, but it could also be used standalone.
It is primarily meant to be used within the Step product, but it could also be used standalone.

"Streaming" here has a slightly special meaning. The purpose is to allow downloads of resources that are
still being produced. We provide a reference implementation based on Websockets.

## Websockets implementation high-level description

1. Upload clients connect to a configurable server endpoint, send a "request upload" message with the file metadata, and expect a "ready for upload" message in return, containing a reference to the endpoint identifying the resource (i.e., a URL/URI). Uploads are currently sent as a one-shot stream upload (i.e., they are not resumable).
2. On the server side, as soon as an upload is initiated, the corresponding resource will be available for download requests, i.e., download clients may connect to the endpoint indicated by the reference, and start requesting data.
3. To provide this functionality, ongoing uploads are checkpointed at regular (configurable) intervals, emitting status updates about the current size of the resource. These status updates are forwarded to all clients interested in the resource in question.
4. A download client connects to the given reference endpoint, and immediately receives the latest known status. It will also receive any potential status updates as soon as they happen on the server side.
5. Clients may at any time request any chunk of the data, identified by a start and end offset, as long as the end offset is within the limits of the last known/checkpointed size on the server.
1. Upload clients connect to a configurable server endpoint, send a "request upload" message with the file metadata, and
expect a "ready for upload" message in return, containing a reference to the endpoint identifying the resource (i.e.,
a URL/URI). Uploads are currently sent as a one-shot stream upload (i.e., they are not resumable).
2. On the server side, as soon as an upload is initiated, the corresponding resource will be available for download
requests, i.e., download clients may connect to the endpoint indicated by the reference, and start requesting data.
3. To provide this functionality, ongoing uploads are checkpointed at regular (configurable) intervals, emitting status
updates about the current size of the resource. These status updates are forwarded to all clients interested in the
resource in question.
4. A download client connects to the given reference endpoint, and immediately receives the latest known status. It will
also receive any potential status updates as soon as they happen on the server side.
5. Clients may at any time request any chunk of the data, identified by a start and end offset, as long as the end
offset is within the limits of the last known/checkpointed size on the server.
6. A typical scenario for a client wishing to retrieve the full file while it is being produced is:
- Wait for new status messages to arrive
- Download the next chunk of data (start offset=offset of already received data, end offset=current file size)
- Terminate once all data is received, as indicated by the status message (`status==COMPLETED`)
7. Of course, it is also possible to retrieve data that has been fully uploaded. In that case, only a single chunk download is actually performed.
8. This implementation has an additional benefit: clients may also request only parts of a file, for instance for client-side pagination of large log files.
- Wait for new status messages to arrive
- Download the next chunk of data (start offset=offset of already received data, end offset=current file size)
- Terminate once all data is received, as indicated by the status message (`status==COMPLETED`)
7. Of course, it is also possible to retrieve data that has been fully uploaded. In that case, only a single chunk
download is actually performed.
8. This implementation has an additional benefit: clients may also request only parts of a file, for instance for
client-side pagination of large log files.

## Project structure
There are currently 7 root-level modules: 4 for the API definitions (common, server, client-upload, client-download), and 2 containing (potentially partial) implementation classes (impl-common, impl-server). The Websocket implementation (impl-websocket) is provided as another module, with submodules mirroring the API modules.

Note that nontrivial Unit tests are consolidated in the websocket server module, because a full "infrastructure" including client and server parts is required for meaningful tests.
There are currently 7 root-level modules: 4 for the API definitions (common, server, client-upload, client-download),
and 2 containing (potentially partial) implementation classes (impl-common, impl-server). The Websocket implementation (
impl-websocket) is provided as another module, with submodules mirroring the API modules.

Note that nontrivial Unit tests are consolidated in the websocket server module, because a full "infrastructure"
including client and server parts is required for meaningful tests.


32 changes: 16 additions & 16 deletions build_parameters.json
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
{
"ACTION": "COMPILE",
"TYPE": "POM",
"FOLDER": ".",
"PARAMETERS": [
{
"NAME": "DEVELOPMENT",
"URL": "nexus-staging::https://nexus-enterprise-staging.exense.ch/repository/staging-maven/",
"CONFIG": "SkipJavadoc"
},
{
"NAME": "PRODUCTION",
"URL": "sonatype::https://obsolete/",
"CONFIG": "SignedBuild"
}
]
}
"ACTION": "COMPILE",
"TYPE": "POM",
"FOLDER": ".",
"PARAMETERS": [
{
"NAME": "DEVELOPMENT",
"URL": "nexus-staging::https://nexus-enterprise-staging.exense.ch/repository/staging-maven/",
"CONFIG": "SkipJavadoc"
},
{
"NAME": "PRODUCTION",
"URL": "sonatype::https://obsolete/",
"CONFIG": "SignedBuild"
}
]
}
Loading