Calculation Engine

The Calculation Engine (CE) is the application that integrates the MUSES modules by providing an API for running scientific calculation workflows involving one or more modules. This document explains how to run the CE locally in Docker Compose and develop its code.

To run the CE in Kubernetes, take a look at the Helm chart driving our production deployment.

Installation

Follow carefully the Docker Compose installation instructions for your operating system.

Clone the git repository from the desired release (replace main with a numbered release tag like 1.2.3 to checkout a specific version):

# Specify the target released version
RELEASE_TAG=1.2.3
# Clone the git repo
git clone -b ${RELEASE_TAG} https://gitlab.com/nsf-muses/calculation-engine.git
# Enter the cloned repo directory for subsequent commands
cd calculation-engine

Configuration

Environment

Customize the configuration as desired by setting environment variable values in a file called .env.dev, following the example in the default values file .env.ce.

A few noteworthy config options:

DEV_MODE: Configure the api-server and celery-worker replicas to restart automatically upon changes to Python source files for rapid code iteration.
FORCE_INITIALIZATION: When running multiple replicas of the api-server, there is a semaphore file .initializing_db that is created to prevent a race condition during database provisioning. In development, you should only run one replica. The FORCE_INITIALIZATION env var prevents an errant initialization from leaving a stale semaphore file that blocks subsequent initializations.

Module registration and manifests

The CE configuration is specified by config/config.yaml. Modules are “registered” for integration into the CE by adding an entry in the config/config.yaml#/modules section of the YAML document.

Private Docker image registry credentials

If there are registered modules with images in private registries, the Docker credentials must be supplied using a standard JSON-formatted string in the DOCKER_CONFIG_JSON env var in a .env.images file. This will populate $HOME/docker/config.json in the Celery worker containers for pulling images from private image registries. Example:

DOCKER_CONFIG_JSON = '{ "auths": { "registry.gitlab.com": { "auth": "dXNlcm5hbWU6YWNjZXNzX3Rva2Vu" }}}'

where the auth value is the output of bash command:

echo -n 'username:access_token' | base64

If the registry is registry.gitlab.com, for example, then you would create a dedicated access token with scopes read_api, read_repository, read_registry.

Start the application

Launch the application with

bash docker/launch.sh dev up

By default you should be able to access the service by opening http://127.0.0.1:4000/ in your browser. Watch logs for the api-server with

bash docker/launch.sh dev logs

You will need to manually run the docker compose logs command if you want to watch other services, such as the Celery workers. Use the command echoed by the launch.sh dev logs script as a template, appending celery-worker for example.

To stop and destroy all containers, use

bash docker/launch.sh dev down

If you need to start with a clean slate (e.g. an empty database), delete the persistent Docker volumes using:

bash docker/launch.sh dev destroy-volumes

You may need to delete the app/calculation_engine/migrations folder when starting fresh since no database migrations are necessary in that case.

Develop CE code

When running locally, you can use the default admin user for testing since the OIDC authentication will not work out-of-the-box. The Python example below shows how to obtain an object for interacting with the CE API as well as an example showing how to list the jobs owned by that user:

from calculation_engine_api import CalculationEngineApi
if __name__ == "__main__":
    api = CalculationEngineApi()
    api.job_list()

Celery tasks can be monitored at http://localhost:4000/flower/tasks. The object storage bucket with files generated by the job can be viewed at http://localhost:9001/browser/calculation-engine.

Users that authenticate via OIDC will not have passwords associated with their Django user accounts. In that case, the user must obtain an API token by visiting http://localhost:4000/, logging in, and then visiting http://localhost:4000/api/v0/token.

Unit tests

The suite of tests can be executing using the command below. Optionally, use the destroy-volumes option to ensure a consistent, empty initial state.

bash docker/launch.sh test destroy-volumes
bash docker/launch.sh test up

This runs the bare minimum services (api-server and database) to execute the run/run_tests.sh script via the run/run_api_server.sh entrypoint script, where the dedicated environment file docker/.env.test sets the TEST_MODE env var.

Staging deployment via Docker Compose

We have a dev/staging deployment running in Docker Compose on a virtual machine hosted on Jetstream2 for rapid collaborative code iteration and testing. It also acts as a staging area for proposed updates before applying the the production instance running in Kubernetes.

The Compose command is different for this staging instance deployment:

bash docker/launch.sh prod up