Skip to content

Update 02_data_transfers.md #266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mdweisner merged 1 commit into from
Jan 16, 2026
Merged

Update 02_data_transfers.md #266

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions docs/hpc/03_storage/02_data_transfers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,34 @@ Please only use OOD for small data transfers! Please use Data-Transfer Nodes (D

### FDT
[FDT](https://fast-data-transfer.github.io/) stands for "Fast Data Transfer". It is a command line application written in Java. With the plugin mechanism, FDT allows users to load user-defined classes for Pre- and Post-Processing of file transfers. Users can start their own server processes. If you have use cases for FDT, visit the [download page](https://github.com/fast-data-transfer/fdt/releases) to get `fdt.jar` to start. Please contact [hpc@nyu.edu](mailto:hpc@nyu.edu) for any questions.

## Copying Data from the Greene HPC Cluster to Torch HPC Cluster
### Copying Data to Torch via the Command Line
Torch has an available data transfer node (DTN) that can be used to upload data. You must authenticate similarly to the steps outlined on the Accessing Torch page.

```
netid@localpc ~ $ rsync -avz -e ssh ~/testfile.txt netid@dtn012.hpc.nyu.edu:/scratch/netid/testfile.txt
(netid@dtn012.hpc.nyu.edu) Authenticate with PIN BUUM5YHJX at https://microsoft.com/devicelogin and press ENTER.
testfile.txt                      100%    0     0.0KB/s   00:00
netid@localpc ~ $
```
## Copying Data to Torch via Globus
Globus has been installed for Torch - you can log in at https://globus.org/ and search for the Torch home, scratch, or archive collections under nyu#torch.​

Globus works much the same as it did on Greene, so see our previous page on how to use it. Once you install Globus Connect Personal on your personal device you can easily transfer files between your local machine and Torch.

## File Transfer (Torch ↔ Greene)
To ease data transfer between clusters, Torch storage has been mounted on the Greene Data Transfer Nodes (e.g. dtn-1.hpc.nyu.edu and dtn-2.hpc.nyu.edu).

This allows users to access the data under /torch:

```
[netid@dtn-1]$ ls /torch
archive home scratch share
```

You can copy files using the rsync command from the Greene DTN:

```
[netid@dtn-1]$ rsync /scratch/netid/my_file.txt /torch/scratch/netid/my_file.txt
```
Loading