Update HPC Client to the Latest Release
Purpose
The HPC Client will receive upgrades with enhancements and fixes. This document describes the process for upgrading the source code for your HPC Client.
Instructions
In the first release of the Flywheel HPC Client, the setup included cloning its git repository. Starting from version 2.1.0, the HPC Client is now available as a pip package. This change allows for easier installation and upgrades. The following instructions will guide you through the process of upgrading the HPC Client to the latest version. Please follow the instructions that best fit your current setup (i.e., whether you installed the HPC Client as a pip package or cloned the repository).
If you encounter any issues, please contact Flywheel staff by either submitting a Zendesk ticket or emailing suppport@flywheel.io for assistance.
Upgrading to the Latest Version using pip (2.1.0 and later)
Starting from version 2.1.0, you can check the version of the HPC Client you have installed by running the command below. If you installed the HPC Client as a user, you won't need to worry about being in a virtual environment to check the version.
- To ensure a smooth upgrade, reserve one hour to upgrade your HPC Client. Do not submit any jobs to the HPC Client, and verify that all previous jobs submitted to the HPC Client have been completed.
- Pause the integration method so it stops running
settings/start-cast.sh
. For example, if Cron was used, use the commandcrontab -e
, comment out thestart-cast.sh
line, and save the changes to the file. - If you will be changing some configuration parameters in your
/settings/cast.yml
, you should create a new branch in with git so that you can easily switch back to your original configuration. To do this, follow the steps below. cd
into your configuration directory repo (e.g.,cd ~/fw-cast
).- Ensure you are in your main/master branch (e.g.,
git checkout master
). - Check the status with
git status
. If you have any uncommitted changes, commit them. - and push any local changes to your private, remote repository.
-
Create and checkout a new branch from your main/master branch; for tracking, include the version number in the name.
```bash git checkout -b ver_<HPC-Client-version> ```
-
Ensure that the python environment where the HPC Client was installed is activated. For example, if you installed it in the base python environment, you can test to see if the fw-hpc-client is installed in that environment by running the command below.
You should see something like
fw-hpc-client==2.1.0
if the HPC Client is installed in that environment. -
Use the command below to upgrade to the latest version. If you installed the HPC Client as a user, you may need to add the
--user
flag to the command. -
If changing the configuration, update the
/settings/cast.yml
file with the new configuration settings. If you are not changing the configuration, you can skip this step. -
Restart the integration method you paused (from step 1.). Monitor casting logs from the HPC Client.
If HPC Client successfully runs, you should get something like the following:
-
Test a gear--like
stress-test
or, if you're testing GPU capabilities,fw-nvidia-cuda-test
-- on your HPC and ensure the HPC Client creates and submits the job. If the gear completes on your HPC without any issues, then the update was successful. If you changed the configuration, commit your changes to your local repository and push them to your remote repository. Create a pull request/merge request to your main/master branch, review it, merge it, checkout your local repository to main/master, then pull these changes. The update is now complete.
Upgrading to the Latest Version from the original, cloned HPC Client repository
Since the latest version uses a pip package, you no longer need the entire repository to run the Flywheel HPC Client. You need only keep the /settings
and /logs
folders; if you're tracking your directory with git, keep your .git
and .gitignore
files.
- To ensure a smooth upgrade, reserve a few hours to upgrade your HPC Client. Stop submit any jobs to the HPC Client, and verify that all previous jobs submitted to the HPC Client have been completed.
- Pause the integration method so it stops running
settings/start-cast.sh
. For example, if Cron was used, use the commandcrontab -e
, comment out thestart-cast.sh
line, and save the changes to the file. -
Ensure that you are tracking your
settings
directory by verifying that it is not in your.gitignore
file. If you wish to not track your/settings/credentials.sh
file in your private, remote repository, you can add it to your.gitignore
file by adding the line below. You should track your/settings/cast.yml
file and/settings/start-cast.sh
file (i.e., don't add these to your.gitignore
file). -
cd
into your configuration directory repo (e.g.,cd ~/fw-cast
). Ensure you are in your main/master branch (e.g.,git checkout master
). Check the status withgit status
. Commit and push any local changes to your private, remote repository. -
Create and checkout a new branch from your main/master branch; for tracking, include the version number in the name.
Example:
-
You have the 3 options listed in "Install hpc-client into your Python environment" in this installation document. The recommended option is to install the HPC Client as to the user site, shown below:
Warning
Do NOT run the setup
command as it will overwrite your current settings.
-
Copy start-cast.sh from this file and replace it with the current one in your
settings
directory. If you have any customizations in your current start-cast.sh, ensure that they are in the new start-cast.sh. If you would like to add GPU capabilities on your HPC Client, follow these steps. -
Restart the integration method you paused (from step 1.). Monitor casting logs from the HPC Client. The logs will now be named
hpc_client.log
instead ofcast.log
.If HPC Client successfully runs, you should get something like the following:
-
Test a gear, like
stress-test
, on your HPC and ensure the HPC Client creates and submits the job. If the gear completes on your HPC without any issues, then the update was successful. It is up to you to remove all the unnecessary files and folders from your repository that were mentioned at the beginning. Commit your changes to your local repository and push them to your remote repository. Create a pull request/merge request to your main/master branch, review it, merge it, checkout your local repository to main/master, then pull these changes. The update is now complete.