Data Transfer

Data Transfer#

This page describes recommended ways to transfer data to and from Aire and Calder, including using command-line tools and Globus.

General guidance#

  • Aire and Calder login nodes provide connections to the campus network, JANET and other universities. Standard tools such as scp and rsync are available for transferring files.

  • You can transfer individual files or directories. For a large number of small files, it may be more efficient to combine them into an archive before transferring them.

  • You can transfer data between HPC and other machines on the University network, including desktop computers, workstations, departmental servers and storage.

Important

Before a job starts, make sure its input data is available in the appropriate HPC storage area. If you need to transfer results elsewhere after a job finishes, write them to scratch during the job and transfer them afterwards.

Warning

Do not transfer data to or from HPC directly from a running compute job unless your workflow specifically requires it. Transfers can keep compute resources occupied while waiting for network operations. Where possible, stage input data before the job starts and transfer output data after it finishes.

Using SCP and rsync#

The command-line tools scp and rsync are available on the login nodes. On Linux, you can use them from a terminal. On Windows, MobaXterm may provide a more convenient interface; on macOS, you could use Cyberduck or ForkLift.

Some graphical SCP clients may prompt for authentication repeatedly. If you need help transferring files with these tools, see the University IT article KB0018323: How to transfer data to and from HPC. You may need to sign in with your University account to view it.

Using Globus#

Globus is the recommended option for transferring large datasets and files reliably between supported storage locations. It provides a web interface for selecting source and destination collections and managing transfers.

For information about University storage options, see the Library’s Storing and handling data guidance. For help transferring data between Globus collections and University storage services, see KB0017543: Data transfer between Globus Collections, OneDrive, Microsoft Teams and SharePoint sites.

Available HPC collection#

The centrally managed Globus collection Aire and Calder - University of Leeds is available for transferring files in users’ home directories. Home directories are shared between Aire and Calder, so the same home-directory files are accessible from either system through this collection. To use it:

  1. Sign in to the Globus web app.

  2. In the Collections panel on the left, search for Aire and Calder - University of Leeds. Turn off all filters if the collection does not appear in the search results.

  3. Select the Aire and Calder - University of Leeds collection, then click Open in File Manager.

  4. In File Manager, double click the users directory, then double click the directory matching your HPC username. This will take you to your home directory, which is shared between Aire and Calder.

  5. Select the files or directories you want to transfer, then choose the destination collection and navigate to the appropriate destination directory.

  6. Start the transfer and monitor its progress in Globus.

Bookmarking the collection

To make future transfers easier, you can bookmark the Aire and Calder - University of Leeds collection. Once bookmarked, you can access it directly from File Manager without having to search for it again.

Scratch storage#

Centrally managed Globus collections for the Scratch filesystems on Aire and Calder are planned, but are not yet available. Until they are enabled, the collection above provides access to home directories only; it does not provide access to Scratch.

If you need to transfer files directly to or from Scratch in the meantime, the existing Globus Connect Personal setup described in the Aire data transfer documentation is a temporary workaround. These setup instructions are not repeated here because the centrally managed collections are the intended approach once Scratch access is available.

Other Globus resources#

If you want to connect Globus to your OneDrive account, you may need to request approval. See Requesting access to applications to connect to your data (M365 Enterprise Apps).

For more information about the Globus service itself, see the Globus file transfer guide.