2 Virtual Lab Setup
It is now time to get yours hands dirty with some hands-on work!
In this chapter you will set up your virtual lab environment in preparation for starting to provision your VMs in the next chapter.
Note
Anticipated time to complete this chapter: TBC from user feedback.
2.1 Install VirtualBox
We will use VirtualBox to serve the VMs for the virtual cluster. The first step in deploying this cluster is to set up the management server (smshost) as a VM which we will then use to deploy the compute nodes.
Remember - this is a virtual training lab!
The virtual management server that we deploy is not required for the final, physical HPC system.
The virtual smshost is a virtualised replica of the final physical management server that will be deployed during a physical HPC system deployment (something that hopefully you will be doing after practicing in this lab).
That said, all steps followed in preparing the virtual smshost will be identical to those followed on the physical management server.
Although this VM can actually be used as the final management server solution for a physical HPC system (with some minor modifications), we do not recommend this as a long-term solution.
-
Download and install VirtualBox
Click here to learn more about VirtualBox.
- VirtualBox is a cross-platform type-2 hypervisor, available as Open Source Software under GPL version 2.
- Supported host operating systems include Windows, Macintosh, Linux and Solaris.
Tip - Stay as close to the same versions as possible!
Always be mindful to maintain the precise version of the software that is outlined in this guide. If no version is explicitly mentioned, then it is safe to assume that the most current version will suffice. That said, where possible, it is advised to use the same versions as were most current at the time of release of OpenHPC 2.x.
Important - package version drifts may lead to unpredictable results. Click here to learn how to avoid this!
While every effort has been taken to ensure software compatibility with users' local environments, if any problems are experienced with the latest version of VirtualBox, it is recommended in that case to make use of one of the verified versions listed in the FAQ that have been used in our test environment.
Once VirtualBox is installed, the framework for running VMs is in place. Although one could now manually create a VM inside the VirtualBox manager GUI (or via the command line), there would be no gaurantee of parity between your VM and the VM expected by the guide.
To ensure configuration parity between the VM on your host machine and the one referenced in this guide, we will use Vagrant to provision the VM framework.
2.2 Install Vagrant
Vagrant provides an automated mechanism to build and maintain VMs across a range of virtualisation hypervisors ('Providers'). We will use Vagrant to configure the smshost VM to precise specifications for the virtual lab (and later to spawn the compute VMs). This ensures that every user will have the same setup, every time.
Click here to learn more about Vagrant and Providers
Vagrant supports various hypervisors ('Providers' in Vagrant), including VirtualBox, Hyper-V, and Docker.
Essentially, this allows a single Vagrantfile to manage VMs in a multitude of virtual environments. For the virtual lab, we have settled on VirtualBox.
For more information, consult the Hashicorp page on Providers, here.
Note
Vagrant manages VM providers, and since VMs are not required for a physical HPC deployment, neither is Vagrant. The use of a virtual environment managed by Vagrant is simply to ensure a reproducible training experience for this virtual lab.
Vagrant is a provisioning tool that ensures that all instances of a vagrant-managed VM are identical upon each instantiation. This parity across systems during training workshops is incredibly useful but is not required for a physical HPC deployment.
-
Match your host machine's OS, not the VM's OS when selecting which Vagrant to download!
Be sure to select the package that matches your host machine's operating system (i.e. the computer that you are using right now), and not the Vagrant package that matches the Rocky Linux VM that we will deploy for the virtual cluster.
Note that your system may need to be rebooted following install.
2.3 Prepare VMs
The file that Vagrant uses to manage and configure the VirtualBox VMs is called a Vagrantfile. We provide you with a preconfigured Vagrantfile, as well as other critical files required for this lab, via Git.
The lab preparation files can be obtained in several ways, but we will show you how to use Git to clone the virtual lab repository which is the recommended method for this virtual lab.
2.3.1 Clone lab repository using Git (recommended)
This option will keep any updates or changes syncronised with your system.
Tip
Updating files in the middle of an OpenHPC deployment that uses Vagrant can cause stabilitiy issues and is not recommended. Nonetheless, when carefully managed, a cloned Git repository is the most suitable approach during a class setting to facilitate live bug fixes or updates to all participants.
1. Download and install Git
Depending on your operating system environment, you will have to choose one of the following options:
- Git BASH for Windows
- Git for Linux
- Git for MacOS
2. Navigate to your preferred Git root for this lab.
For example: <your_home_directory>/openhpc-2.x-virtual-lab/
If the example is unclear, click here to understand what the Git root should look like.
The Git root will be the base file directory for the virtual cluster files, for example:
/home/me/openhpc-2.x-virtual-lab/
or
c:/users/me/Desktop/openhpc-2.x-virtual-lab/
Note
For the purposes of this guide the Git root will be assumed to be ~/openhpc-2.x-virtual-lab/ although your actual path may differ.
The sample syntax [git root]# can be interpreted as a prompt at ~/openhpc-2.x-virtual-lab/ and you can substitute this syntax for the path that you have chosen for your Git root.
3. Clone the Git repository
The public repository for the virtual lab can be cloned using HTTPS:
[<your git root>]# [<your git root>]# |
Click here to understand what git clone does.
The command git clone will replicate a public Git repository to a local directory on your computer.
There are two methods for cloning a Git repository - SSH and HTTPS. By cloning via HTTPS we are doing a one-way download of the files that are present on the Git repository.
If there are any future updates to the repository, these can be pulled down to the local copy using git pull.
4. View lab files
The files you have cloned from the remote Git repository are now available on your local machine at the location <git_root>/openhpc-2.x-virtual-lab/.
The most important files at present are:
| File | Description |
|---|---|
input.local.lab |
The configuration parameter script for the virtual lab. Adapted from the original OpenHPC file input.local |
Vagrantfile |
The Vagrant configuration file for setting up the lab virtual machines |
compute-node.box |
A pre-configured Vagrant box for setting up the compute VMs |
slurm.conf.lab |
The customised Slurm configuration file for the virtual lab |
Click here for a recap of the chapter.
In this chapter the virtual lab environment was set up:
-
VirtualBox - hypervisor used to host the virtual machines in the virtual cluster
-
Vagrant - used to manage the configuration of virtual machines within VirtualBox
-
Vagrantfile- the file used by Vagrant to define the parameters for the VMs used in the virtual lab
Congratulations
Your virtual lab envionment is ready to deploy the virtual cluster! Let's move on to configuring your management node in Chapter 3.