UCSD Research Cluster: User Guide

This documentation includes guidance, instructions, general information about the Research IT Service managed Research Cluster. Researcher execute computing jobs on the cluster in containerized environments known as Docker “containers” which are essentially lightweight virtual machines, each assigned dedicated CPU, RAM, and GPU hardware, and each well isolated from other users’ processes. The Research Cluster uses the Kubernetes container management/orchestration system to route users’ containers onto compute nodes, monitors performance, and applies resource limits/quotas as appropriate.

Complex Machine Learning workflows are supported through terminal/SSH logins and a full Linux/Ubuntu CUDA development suite. Users may install additional library packages (e.g. conda/pip, CRAN) as needed, or can opt to replace the default environment entirely by launching their own custom Docker containers. High-speed cluster-local storage houses workspaces and common training corpora (e.g. CIFAR, ImageNet).

Note to Users:

When using the Research Cluster, please be considerate and terminate idle containers prior to closing you command line interface or logging out of the datahub. When a user engages a container, the container become unusable by others even if completely idle. While containers share system RAM and CPU resources under the standard Linux/Unix model, the cluster’s 80 GPU cards are assigned to users on an exclusive basis.

[ 1 Getting Started ] [ 2 Launching a Container ] [ 3 Web Interface Tool ] [ 3.1 Monitoring Container Resource Usage ] [ 4 Modifying Containers ] [ 4.1 Container Run Time Limits ] [ 4.2 Container Termination Messages ] [ 5 Data Storage / Datasets ] [ 5.1 Standard Datasets ] [ 6 File Transfer ] [ 7 Customizing a Container Environment ] [ 7.1 Adjusting Container Environment and CPU/RAM/GPU limits ] [ 7.2 Alternate Docker Images ] [ 7.3 Adjusting Launch Script Environments Command Line Options ] [ 7.4 Custom Python Packages (Anaconda/PIP) ] [ 7.5 Installing TensorBoard ] [ 7.6 Running Jobs in a Background Container and Long-Running Jobs ] [ 8 Run-TIme Error Messages ] [ 9 Monitoring Cluster Status ] [ 10 Cluster Hardware Specifications ] [ 11 Licensed Software ] [ 12 Acknowledging Research IT Services ]

Getting Started

The are two ways in which to access the Research Cluster - via SSH or via the Datahub.

First, login via SSH to the “researchcluster-login.ucsd.edu" Linux server login node using your UC San Diego Active Directory (AD) username (with ‘@researchcluster-login.ucsd.edu’) and password. After logging in, you will be in a login node for the Research Cluster and should not perform any computation in the login node.

Login step-by-step guidance:

Open command line interface - known as the 'Terminal' for MacOS and 'Command Prompt' for Windows.
Enter command ‘ssh ADusername@researchcluster-login.ucsd.edu'.
1. If the researchcluster-login.ucsd.edu login node is currently down/unavailable, you may TEMPORARILY use the “dsmlp-login.ucsd.edu” login node instead. The dsmlp-login.ucsd.edu login node is intended for instructional class work use by students as its first priority, so it’s recommended to always use the researchcluster-login.ucsd.edu for research work. Student involved in research projects should login to the relevant login node depending on the intended work (class work versus research work).

You may be asked a question after entering your username. Select 'yes’ to continue connecting.

Enter your password. Note: Your password will not display as you enter it.
A successful login will display your last login information. For example, ‘Last login: Thu Aug 3 10:25:19 2023 from 137.110.14.162’. Note that you are now in the Login Node.

IMPORTANT: DO NOT RUN JOBS IN THE LOGIN NODE! Jobs must only be run in a launched container. Follow the guidance in the next section (Launching a Container), before running your compute jobs.

Login page: https://datahub.ucsd.edu .
- Enter UCSD email address and AD password. You will be sent a push to confirm via Duo.
Select the your chosen notebook environment.
- Research Cluster users will have a choice of multiple environments to select.
- If joining a PI/lab specific environment, your may only see the name of your PI/lab’s environment.
- The ‘public’ folder will include storage that is shared and where datasets can be stored for all to access.
Next, you’ll be directed to your environment and see the Jupyter Notebook interface.
Click ‘New’ and select the kernel you wills to start up.
When done using the datahub, select ‘logout’ to terminate kernels and end session.

Launching a Container

After signing into the login node, you can start a pod/container using launching a standard Research Cluster launch script or a customize container launch script.

Once started, containers are accessible in either a Bash Shell (command-line) or a Jupyter/Python Notebook environment. Users may access their Jupyter notebook by copying and pasting the launch script link provided by pasting the link in the browser address bar. This link will work as long as your container is active and will cease to work once you logout. Docker container image and CPU/GPU/RAM settings are all configurable - see the “Customization” and "Launch Script Command-line Options" sections below for more details.

Containers terminate when automatically when users exit the interactive shell.

More details and guidance on launching a container is available on the “How To: Launching Containers From the Command Line - Data Science/Machine Learning Platform (DSMLP)” guidance page.

A successfully launched container will display similar output as follows:

Note in the example, the link provided to access the Jupyter notebook is “http://dsmlp-login.ucsd.edu:9572/user/kkt008/?token=8f7e5fbfa093f5a29fe115ca8b24f1fc03295c4d26886a83facbfb8750e02698”.

The standard launch scripts are predefined meaning they have specific RAM and CPU (and/or GPU) configurations. Other launch scripts are available at /software/common64/dsmlp/bin/.

Launch Script	Description	#GPU	#CPU	RAM	Container Image(s)

Launch Script	Description	#GPU	#CPU	RAM	Container Image(s)
launch-scipy-ml.sh	Python 3, PyTorch, TensorFlow	0	2	8	ucsdets/scipy-ml-notebook:2020.2.9
launch-scipy-ml-gpu.sh	Python 3, PyTorch, TensorFlow	1	4	16	ucsdets/scipy-ml-notebook:2020.2.9
launch-datascience.sh	Python 3, Datascience, R	0	2	8	ucsdets/datascience-notebook:2020.2-stable
launch-rstudio.sh	R-Studio	1	4	16	ucsdets/datascience-rstudio:latest

Standard images with ‘pytorch’ include the GNU Screen utility, which may be used to manage multiple terminal sessions in a window-like manner.

Web Interface Tool

The Research Cluster uses the web interface tool known as Jupyterhub Notebooks as an alternative graphical interface option for users who prefer computing in this type of interface rather than in the command line interface.

To access the web interface tool, users are directed to sign-in at https://datahub.ucsd.edu/ (or via selecting the login button up top).

Web-based Jupyter notebooks allow researchers to combine live code, equations, visualizations and narrative text for data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and more. The Research Cluster's Jupyter notebooks offer straightforward interactive access to popular languages and GPU-enabled frameworks such as Python, R, Pandas, PyTorch, TensorFlow, Keras, NLTK, and AllenNLP.

The default container configuration creates an interactive web-based Jupyter/Python Notebook which may be accessed via a TCP proxy URL output by the launch script. Note that access to the TCP proxy URL requires a UCSD IP address: either on-campus wired/wireless, or VPN. See http://blink.ucsd.edu/go/vpn for instructions on the campus VPN.

Monitoring Container Resource Usage

Cluster users can monitor their resource usage in a launched container in both the command line terminal and in the web interface tool. Monitoring resource usage allows for users to be aware of their job limitations, as well as, identify possible bottlenecks during certain stages of the job execution.

Modifying Containers

Certain modifications can be made to containers to allow for users to adjust their environment to accommodate specific computing needs.

Container Run Time Limits

Container Termination Messages

Data Storage / Datasets

There are two types of persistent file storage are available within containers - private/home directory and shared directory storage.

A private home directory ($HOME) for automatically generated for each cluster user. User's private home directory is limited to a 100GB storage allocation by default.
A shared directory - for group shared data or for datasets used to distribute common data (e.g. CIFAR-10, Tiny ImageNet) for individual access. Shared directory storage can vary as this storage may be a mounted storage.

In specific cases, Research IT may make allowances to temporarily increase storage in a user’s private home directory. These requests may be submitted by emailing rcd-support@ucsd.edu.

Standard Datasets

File Transfer

Users can utilize commands (e.g. 'git', 'scp', 'sftp', and 'curl') in the bash shell (command line interface) to import code or data from external servers that are both on and off-campus. Files can be copied into the cluster from external sources using Globus, SCP/SFTP, or RSYNC.

Customizing a Container Environment

Each launch script specifies the default Docker image to use, the required number of CPU cores, GPU cards, and GB RAM assigned to a container. When creating a customized container, it is recommended to use CPU-only containers until your code is fully tested and a test training run has been completed successfully. It is important to note that PyTorch, Tensorflow, and Caffe toolkits can easily switch between CPU and GPU , as such, a successful run in a CPU-only container should also be successful in a container with GPU.

Adjusting Container Environment and CPU/RAM/GPU limits

All running containers in the cluster have a maximum configuration limit of 8 CPU, 64GB, and 1 GPU. You may run eight 1 CPU-core containers, one 8-core container, or any configuration within the these bounds. Requests may be submitted to rcd-support@ucsd.edu to increases to these default limits, as well as, to request other adjustments (including software) to your container environment.

Alternate Docker Images

In addition to configuration settings, users can import alternate or custom Docker images. The cluster servers will pull container images from dockerhub.io or elsewhere if requested. You can create or modify these Docker images as needed.

Adjusting Launch Script Environments Command Line Options

Users can change the default variables within a launch script environment variables using specific command line options.

Custom Python Packages (Anaconda/PIP)

Users may install additional Python packages within their containers using the PIP tool or standard Anaconda package management system. Users should only install Python packages after launching a container. When Python packages are installed, they are installed in a user’s home directory. As such, these packages will be available for all containers launched thereafter by the user.

For less complex installations the PIP tool can be used to install Python packages. Please see PIP documentation ‘User Installs’ for detailed guidance.
Anaconda is recommended for installing scientific packages with complex dependencies. Please see Anaconda's Getting Started for a guided introduction.

Installing TensorBoard

Current cluster configuration does not permit easy access to Tensorboard via port 6006; however, there are shell commands that can install a TensorBoard interface accessible within the Jupyter environment.

Running Jobs in a Background Container and Long-Running Jobs

To minimize the impact of abandoned/runaway jobs, the cluster allows for containers to run jobs in the background container for up to 12 hours of execution time. Users need to specify that a job should run in a background container by using the "-b" command line option (see example below). To support longer run times, the default execution time can be extended upon request to rcd-support@ucsd.edu.

Note to users: Please be considerate and terminate any unused background jobs. GPU cards are limited and assigned to containers on an exclusive basis. When attached to a container, GPUs are unusable by others even if the GPU is idle while attached to your container.

Run-TIme Error Messages

There may be instances where you receive a CUDA run-time error while running a job in a container. Below are a few of the more commonly encountered errors. These errors can typically be resolved by user adjustments. However, If users encounter a run-time error that requires more assistance to resolve, please contact rcd-support@ucsd.edu.

Monitoring Cluster Status

Users can enter the ‘cluster-status’ command for insight into the number of jobs currently running and GPU/CPU/RAM allocated. Alternatively, users can refer the the cluster ‘Node Status’ page for updates on containers (or images).

Cluster Hardware Specifications

The Research Cluster shares hardware infrastructure with the Data Science and Machine Learning Platform (DSMLP). As such, the information about the hardware specifications for the Research Cluster are described in the Cluster architecture diagram (as displayed in reference to the DSMLP).

Licensed Software

Installing licensed software is allowed in the Research Cluster; however, certain software versions are required to be compatible for installation in a cluster environment. The purchase of licensed software is the responsibility of the user or their sponsoring department. Research IT Services is available to assist with the installation of licensed software. For questions about installing licensed software, please email rcd-support@ucsd.edu.

Acknowledging Research IT Services

Papers, presentations, and other publications that feature research that benefited from the Research Cluster computing resource, services or support expertise may include in the text the following acknowledgement:

This research was done using the UC San Diego Research Cluster computing resource, supported by Research IT Services and provided by Academic Technology Services / IT Services.