Tranferring Data

There are several recommended approaches for transferring data to and from the UConn Health HPC file system. Each is most suitable for different situations. The table below summarizes the best use case for each tool.

Tool Intended Use Case
Secure Copy (scp) Transfers of less than < 10 GB. Command line interface and scripting.
Globus Large data transfers of > 10 GB. Highly robust. Recommended for sharing outside of UConn. Interactive user interface and command line interface options
SMB Share Accessing modestly sized files directly from applications running locally.

Secure Copy

Secure copy is ideal for transferring small amounts of data. A secure copy can be performed with the scp command which is available in the Windows PowerShell or MacOS / Linux Terminal.

This command should only be used to transfer data to and from a transfer node which is a dedicated node for transferring data. The transfer nodes have accesses to the same file systems as the login and compute nodes. We recommend using the transfer.hpc.cam.uchc.edu URL. Transferring data through this URL requires that you have setup SSH keys and that you be connected to the UConnHealth-SECURE WiFi or the CAM VPN.

Transferring a File To the UConn Health HPC File System

In order to transfer data from a local computer to the UConn Health HPC file system with scp:

scp <local_file_path> <user>@transfer.hpc.cam.uchc.edu:<remote_path>

Where <local_file_path> is the path to the file on your local computer, <user> is your CAM username, and <remote_path> is the path on the UConn Health HPC file system where you want to copy the file. <remote_path> can be excluded if you want to copy the file to your home directory.

Transferring a File From the UConn Health HPC File System

To transfer data from the UConn Health HPC file system to a local computer with scp:

scp <user>@transfer.hpc.cam.uchc.edu:<remote_file_path> <local_file_path>

Where <user is your CAM username, <remote_file_path> is the path to the file on the UConn Health HPC file system, and <local_file_path> is the path on your local computer where you want to copy the file. Simply use a . for <local_file_path> to copy the file to your current working directory.

Transferring a Directory

In order to transfer a directory, you will need to use the -r flag to recursively copy all files and subdirectories. For example, to copy a directory from your local computer to the UConn Health HPC file system:

scp -r <local_directory_path> <user>@transfer.hpc.cam.uchc.edu:<remote_path>

Wildcards

Secure copy also supports the use of wildcards such as * for the remote path as long as wildcard characters are escaped with a \ or the url + path are enclosed in quotes. For example, to copy all .txt files from your UConn Health HPC file system home directory to your local computer:

scp <user>@transfer.hpc.cam.uchc.edu:\*.txt <local_directory_path>

or

scp "<user>@transfer.hpc.cam.uchc.edu:*.txt" <local_directory_path>

Alternative Transfer Host

There is an additional host that can be used to transfer data without an SSH key. This host requires your CAM password for authentication if you have not setup an SSH key.

The URL for this host is transfer.cam.uchc.edu.

Globus

Globus is well suited for large file transfers, sharing files, and for transferring between institutions such as when you need to share files with a collaborator.

Getting Started

Globus has excellent documentation. We recommend starting with this tutorial which contains almost everything you need to know to use Globus in addition to making note of the specifics below.

Accessing the UCHC Globus Server

Organizational Login

When first logging in to the Globus App, you will be prompted to select your organization. You should select the University of Connecticut. If you are not already logged in, you will be prompted to enter your UConn NetID and password.

UCHC Globus Server Collection

To access the UConn Health HPC file system, search for and select, the UCHC Globus Server in the collection text field. Upon selecting the UCHC Globus Server, you will be prompted to enter your CAM username and password.

File Paths

Navigation to your home directory mirrors the cluster file system. You can access your home directory by entering /home/FCAM/<user> in the path text field.

Navigation to other directories has one minor but unintuitive difference. The root of the cluster file system is /globus instead of /. So, for example, to access the /seqdata directory you would enter /globus/seqdata in the Path text field.

Transferring Files Locally

In order to transfer files to a local computer, you will need to install the Globus Connect Personal application.

Sharing Files

Files can be shared with anyone that has a Globus account including collaborators at other institutions. For a guide on sharing files with Globus, , see this tutorial.

Note

When adding permissions, you generally will not need to modify the Path text field. The / in this case refers to the root of the collection and not the UCHC HPC file system.

Globus Command Line Interface

The Globus Command Line tool is available as a module that can be loaded into your environment with module load globus-cli. See the Globus CLI documentation for more information on using the command line interface.

Mounting SMB Share

SMB is a network file sharing protocol that allows applications such as your file manager to read and write to files on a remote server as if they were local files. IGV or your browser are other examples of applications which you may want to use to read files directly on the UConn Health HPC file system.

To connect to the UConn Health HPC file system using SMB and mount one of the directories, you will need to determine the URL or IP address and path of the directory you want to mount. Below is a list of the most commonly used directories and their URLs or IPs.

Directory Windows MacOS / Linux
/home/FCAM/home \\cfs09.cam.uchc.edu\home\FCAM\<user> smb://cfs09.cam.uchc.edu/home/FCAM/<user>
/labs \\cfs15.cam.uchc.edu\labs\<path> smb://cfs15.cam.uchc.edu/labs/<path>
/projects \\155.37.251.38\projects\<path> smb://155.37.251.38/projects/<path>
/departments \\155.37.250.151\departments\<path> smb://155.37.250.151/departments/<path>
/scratch \\cfs08.cam.uchc.edu\ifs\scratch\<path> smb://cfs08.cam.uchc.edu/ifs/scratch/<path>
/sandbox \\155.37.248.191\sandbox\<path> smb://155.37.248.191/sandbox/<path>

Replace <user with your CAM usern<path> with the specific subdirectory path that you want to access.

The df -h command can be used to list all of file systems and their URLs.

Connecting

To connect to the UConn Health HPC file system using SMB, you will need to follow the instructions below for your specific operating system.

  1. Connect to the CAM VPN if not on the UConnHealth-SECURE WiFi network.
  2. Open File Explorer from the taskbar or the Start menu.
  3. Select This PC from the left pane. Then, on the File Explorer ribbon, select More > Map network drive.

  1. In the Drive list, select a drive letter. (Any available letter will do.)
  2. In the Folder box, enter the SMB server URL and path you want to mount from the table above. To connect every time you sign in to your PC, select Reconnect at sign-in.
  3. Select Finish.
  4. When prompted for a username and password, enter your username as CAM<user> and enter your CAM password.

  1. Connect to the CAM VPN if not on the UConnHealth-SECURE WiFi network.
  2. Open File Explorer from the taskbar or the Start menu
  3. Select This PC from the left pane. Then, on the Computer tab, select Map network drive.

  1. In the Drive list, select a drive letter. (Any available letter will do.)
  2. In the Folder box, enter the SMB server URL and path you want to mount from the table above. To connect every time you sign in to your PC, select Reconnect at sign-in.
  3. Select Finish.
  4. When prompted for a username and password, enter your username as CAM<user> and enter your CAM password.

  1. Connect to the CAM VPN if not on the UConnHealth-SECURE WiFi network.
  2. Open Finder and select Go:Connect to Server from the top menu.
  3. Enter the address of the SMB server URL and path you want to connect to.
  4. In order to automatically mount the SMB share, navigate to System Settings > General > Login Items & Extensions.
  5. Under the Open at Login section, click the + button and

  1. Select the mounted SMB share on the sidebar and then the directory or directories in your finder window.

  1. When prompted for a username and password, enter your CAM username and password.

FileZilla

FileZilla is a free and open-source FTP client that supports SFTP. It is available for Windows, MacOS, and Linux. It is suitable for transferring dozens of files or tens of GB of data.

Usage

  1. Setup SSH keys.
  2. Open FileZilla and select File:Site Manager from the top menu.
  3. Click New Site and enter a name for the site.
  4. Select SFTP - SSH File Transfer Protocol as the protocol.
  5. Enter transfer.hpc.cam.uchc.edu as the host.
  6. Select Key file as the logon type.
  7. Enter your CAM username in the User field.
  8. Enter or browse to the path of your private key file. This is the file that you generated when you setup SSH keys.