Skip to content

5 - Development guide

The basic structure of the python package is as follows:

pyproject.toml         \   
poetry.lock             >  Package management
poetry.toml            /
hpc_client/
├── cast.py            Main script
│
├── assets
│   ├── cast.yml       Default hpc-client settings
│   ├── credentials.sh Default Credentials environment file
│   └── start-cast.sh  Default start cast script
│ 
├── cluster
│   ├── __init__.py    Cluster type disptach
│   │
│   ├── base.py        Common override points
│   ├── common.py      Fixed functionality; do not modify
│   │
│   ├── lsf.py         \
│   ├── sge.py          >  Off the shelf implementations
│   └── slurm.py       /
│
└── util               Fixed functionality; do not modify
    ├── __init__.py
    ├── basic_setup.py Basic setup of configuration directory
    ├── defn.py        Data structure definitions
    ├── frame.py       Program setup
    └── net.py         SDK helpers

Adding a new queue type to the hpc-client

To add a new queue type, follow these steps:

  1. Make a copy of cluster/slurm.py and name it after your queue type.

  2. Edit cluster/__init__.py so it knows about your new class.

  3. Edit your new file's defaults to what makes sense for your use case.

  4. Review cluster/base.py for the various methods you can override.

  5. Avoid modifying or overriding the functions in cluster/common.py.

    We reserve the right to make changes to those functions for future Flywheel features.

  6. Avoid modifying the util package. Instead, place helper functions on your new class.

  7. Build and Install a new fw-hpc-client package

    poetry build
# Install the package into your environment e.g.:
pip install --user ./dist/fw-hpc-client-<version>.whl