Update HPC Client to the Latest Release

Purpose

The HPC Client will receive upgrades with enhancements and fixes. This document describes the process for upgrading the source code for your HPC Client.

Instructions

In the first release of the Flywheel HPC Client, the setup included cloning its git repository. Starting from version 2.1.0, the HPC Client is now available as a pip package. This change allows for easier installation and upgrades. The following instructions will guide you through the process of upgrading the HPC Client to the latest version. Please follow the instructions that best fit your current setup (i.e., whether you installed the HPC Client as a pip package or cloned the repository).

If you encounter any issues, please contact Flywheel staff by either submitting a Zendesk ticket or emailing suppport@flywheel.io for assistance.

Upgrading to the Latest Version using pip (2.1.0 and later)

Starting from version 2.1.0, you can check the version of the HPC Client you have installed by running the command below. If you installed the HPC Client as a user, you won't need to worry about being in a virtual environment to check the version.

fw-hpc-client --version

To ensure a smooth upgrade, reserve one hour to upgrade your HPC Client. Do not submit any jobs to the HPC Client, and verify that all previous jobs submitted to the HPC Client have been completed.
Pause the integration method so it stops running settings/start-cast.sh. For example, if Cron was used, use the command crontab -e, comment out the start-cast.sh line, and save the changes to the file.
If you will be changing some configuration parameters in your /settings/cast.yml, you should create a new branch in with git so that you can easily switch back to your original configuration. To do this, follow the steps below.
cd into your configuration directory repo (e.g., cd ~/fw-cast).
Ensure you are in your main/master branch (e.g., git checkout master).
Check the status with git status. If you have any uncommitted changes, commit them.
and push any local changes to your private, remote repository.
Create and checkout a new branch from your main/master branch; for tracking, include the version number in the name.
```
```bash
git checkout -b ver_<HPC-Client-version>
```
```
Ensure that the python environment where the HPC Client was installed is activated. For example, if you installed it in the base python environment, you can test to see if the fw-hpc-client is installed in that environment by running the command below.
```
pip freeze | grep fw-hpc-client
```
You should see something like fw-hpc-client==2.1.0 if the HPC Client is installed in that environment.
Use the command below to upgrade to the latest version. If you installed the HPC Client as a user, you may need to add the --user flag to the command.
```
pip3 install --upgrade fw-hpc-client
```
If changing the configuration, update the /settings/cast.yml file with the new configuration settings. If you are not changing the configuration, you can skip this step.

Restart the integration method you paused (from step 1.). Monitor casting logs from the HPC Client.

tail -f logs/hpc_client.log

If HPC Client successfully runs, you should get something like the following:

07-25 21:35:02 DBUG Looking for jobs to cast...
07-25 21:35:02 DBUG Found 0 jobs in 151 ms.
07-25 21:35:02 INFO No jobs to handle. Runtime: 413 ms.

Test a gear--like stress-test or, if you're testing GPU capabilities, fw-nvidia-cuda-test-- on your HPC and ensure the HPC Client creates and submits the job. If the gear completes on your HPC without any issues, then the update was successful. If you changed the configuration, commit your changes to your local repository and push them to your remote repository. Create a pull request/merge request to your main/master branch, review it, merge it, checkout your local repository to main/master, then pull these changes. The update is now complete.

Upgrading to the Latest Version from the original, cloned HPC Client repository

Since the latest version uses a pip package, you no longer need the entire repository to run the Flywheel HPC Client. You need only keep the /settings and /logs folders; if you're tracking your directory with git, keep your .git and .gitignore files.

To ensure a smooth upgrade, reserve a few hours to upgrade your HPC Client. Stop submit any jobs to the HPC Client, and verify that all previous jobs submitted to the HPC Client have been completed.
Pause the integration method so it stops running settings/start-cast.sh. For example, if Cron was used, use the command crontab -e, comment out the start-cast.sh line, and save the changes to the file.
Ensure that you are tracking your settings directory by verifying that it is not in your .gitignore file. If you wish to not track your /settings/credentials.sh file in your private, remote repository, you can add it to your .gitignore file by adding the line below. You should track your /settings/cast.yml file and /settings/start-cast.sh file (i.e., don't add these to your .gitignore file).
```
/settings/credentials.sh
```
cd into your configuration directory repo (e.g., cd ~/fw-cast). Ensure you are in your main/master branch (e.g., git checkout master). Check the status with git status. Commit and push any local changes to your private, remote repository.
Create and checkout a new branch from your main/master branch; for tracking, include the version number in the name.
```
git checkout -b ver_<HPC-Client-version>
```
Example:
```
git checkout -b ver_2.1.0
```

You have the 3 options listed in "Install hpc-client into your Python environment" in this installation document. The recommended option is to install the HPC Client as to the user site, shown below:

pip3 install --user fw-hpc-client
# Ensure $HOME/.local/bin is in your $PATH
echo $PATH
# If not, add it to your .bashrc file
echo "export PATH=\$PATH:\$HOME/.local/bin">>$HOME/.bashrc
# Update your environment
source $HOME/.bashrc

Warning

Do NOT run the setup command as it will overwrite your current settings.

Copy start-cast.sh from this file and replace it with the current one in your settings directory. If you have any customizations in your current start-cast.sh, ensure that they are in the new start-cast.sh. If you would like to add GPU capabilities on your HPC Client, follow these steps.
Restart the integration method you paused (from step 1.). Monitor casting logs from the HPC Client. The logs will now be named hpc_client.log instead of cast.log.
```
tail -f logs/hpc_client.log
```
If HPC Client successfully runs, you should get something like the following:
```
07-25 21:35:02 DBUG Looking for jobs to cast...
07-25 21:35:02 DBUG Found 0 jobs in 151 ms.
07-25 21:35:02 INFO No jobs to handle. Runtime: 413 ms.
```
Test a gear, like stress-test, on your HPC and ensure the HPC Client creates and submits the job. If the gear completes on your HPC without any issues, then the update was successful. It is up to you to remove all the unnecessary files and folders from your repository that were mentioned at the beginning. Commit your changes to your local repository and push them to your remote repository. Create a pull request/merge request to your main/master branch, review it, merge it, checkout your local repository to main/master, then pull these changes. The update is now complete.